AI Legal Judgment Prediction Prone to Shortcut Learning, Study Finds.

New research investigates a critical limitation in Legal Judgment Prediction (LJP) systems: shortcut learning. The study, using 33,158 UK Employment Tribunal claims, found that AI models, from simple TF-IDF classifiers to advanced LLMs, often achieve high predictive performance by identifying outcome-revealing linguistic cues embedded in judicial texts that are written after the judgment. This means models might be retrospectively classifying rather than genuinely forecasting. The findings highlight that performance figures can be inflated by these "leakage" features. A model trained on just 4% of identified leakage features could even outperform human experts. However, the research also offers a path forward: actively auditing and masking these contaminated texts. When models were retrained after removing these shortcut features, the reduction in predictive performance was minimal, suggesting that LJP systems are still capable of extracting meaningful predictive signals from the core content. This implies that while AI will exploit shortcuts if available, careful data preparation and auditing can lead to more reliable and genuinely predictive legal AI tools.