Contrastive Reflection Optimizes LLM Prompts for Agentic IR Workflows

Optimizing prompts for large language model (LLM) agents, particularly in information retrieval (IR) workflows, often resembles a debugging process more than a blind search. Engineers require clear insights into specific failures, an understanding of what distinguishes successful from unsuccessful behaviors, and a reliable way to ensure prompt edits genuinely improve quality without introducing regressions. This paper presents Contrastive Reflection, an iterative framework designed to address these precise needs. Contrastive Reflection begins with a task-centric definition of quality, leveraging structured traces from agents—such as retrieval or reasoning paths from QA agents, or dimension-level scores and rationales from grading agents. These traces are used to pinpoint error-anchored behavioral slices, which are then contrasted with nearby successful examples from the same operational region. A 'Teacher LLM' is subsequently employed to propose targeted prompt edits based on these contrasts. Crucially, candidate edits are only accepted if they demonstrate improved validation performance, with optional checks to prevent regressions, thereby ensuring a robust and interpretable optimization loop. The framework achieved significant accuracy improvements on a public HotpotQA retrieval-augmented QA setup, outperforming several modern prompt optimizers.