Agentic Context Learning Improves LLM Task Success

Large Language Models (LLMs) often struggle with "context learning," a task where they must learn and apply new, task-specific knowledge from complex contexts not present in their pre-training. Despite extensive research, even frontier models achieve low success rates. This study empirically investigates the root causes of these failures. Contrary to the common hypothesis that failures are due to content access issues, the research found that the primary challenge lies in acquiring local specifications. These specifications include domain-specific formats, local rules, and completeness conditions, which are often implicitly distributed throughout the context rather than explicitly stated in the query. Analysis of a large benchmark, CL-Bench, revealed that over half of evaluation criteria relate to specification acquisition, while only a quarter relate to content acquisition. To address this, the researchers developed a simple intervention called Private Specification-Contract Induction (PSCI). PSCI works by extracting these local specifications and then enforcing them through adversarial checking and repair mechanisms. This method significantly improved performance, achieving state-of-the-art results on CL-Bench with various LLMs, demonstrating that context learning heavily relies on the ability to not only retrieve information but also to infer and apply the underlying rules and requirements.