CDR-Bench Reveals LLMs Fail Compositional, Order-Sensitive Data Refinement.

Data refinement tasks often involve executing multiple sequential operations on evolving text states, where both the combination and the order of these operations are crucial for the final outcome. While existing benchmarks have explored text editing or combined it with code execution, it has been unclear whether large language models (LLMs) can directly and reliably perform these complex, compositional, and order-sensitive data refinement recipes. To address this gap, researchers introduced CDR-Bench, a comprehensive benchmark comprising 3,462 high-quality tasks across four real-world data refinement domains and 29 distinct operators. The benchmark evaluates LLMs in atomic, order-agnostic, and order-sensitive settings, using deterministic reference outputs for precise evaluation. Experiments conducted with over ten state-of-the-art LLMs revealed consistent failure patterns. Model performance sharply declined in compositional settings, and success rates for order-sensitive recipes collapsed. These findings underscore a significant deficiency in current LLMs regarding the procedural faithfulness required for dependable compositional data refinement, indicating a need for further research in this area.