LLMs Struggle with Temporal Legal Reasoning, Applying Wrong Laws

Large Language Models (LLMs) face significant challenges in legal reasoning, particularly when it comes to identifying and applying the correct version of a law based on its effective date. This capability, termed "temporal applicable-law determination," is critical for tasks like legal judgment prediction. Researchers developed a benchmark to assess LLMs on this specific skill, uncovering several key issues. The study found that LLMs consistently favor the most recently enacted law, irrespective of when the relevant facts of a case occurred. This bias is not due to a lack of understanding about laws having temporal scopes or a deficit in knowledge of historical statutes. Instead, the research suggests that reinforcement learning, while enhancing general reasoning, might inadvertently reduce the diversity of reasoning paths, leading models to converge on applying only current law. Counterintuitively, models with superior general reasoning abilities often performed worse on these temporal legal reasoning tasks, highlighting a specific blind spot in their capabilities.