LLMs Struggle with Many Simultaneous Instructions, Study Finds.

Researchers introduced the Constraint Saturation Evaluation (CSE) benchmark to systematically assess how large language models perform when faced with multiple simultaneous instructions. The study involved 15 models, 36 constraint types, and nearly 370,000 checks, with results verified deterministically without LLM judges. The findings indicate that while LLMs can handle individual constraints well, their overall success rate in satisfying *all* constraints drops dramatically as the number of simultaneous instructions increases. For instance, a model passing individual constraints at 41% with eight constraints only succeeded on all eight 5.7% of the time. This "phase transition" shows a rapid collapse in compositional constraint satisfaction. Furthermore, the study found that different constraint types degrade unequally, with structural constraints (e.g., reasoning structure) losing capability twice as fast as lexical ones. Failures were largely independent, accumulating multiplicatively, suggesting that the models struggle with sustained tracking rather than pairwise interference between instructions. Reliable instruction following typically breaks down beyond 5-6 simultaneous constraints for even the strongest models.