SAGE Improves Dialogue Agent Evaluation with State-Grounded Metrics

Evaluating task-oriented dialogue agents presents a unique challenge: it's not enough for a reply to sound natural; it must also correctly progress the underlying workflow state. Conventional LLM-as-a-judge methods often miss this crucial distinction because they assess the entire conversation context as a single unit and require expensive, full-model calls for each turn. To address these limitations, SAGE (State-Grounded Abstention-Aware Evaluation) has been proposed. This method compiles a workflow specification and per-turn state differences into atomic, schema-grounded criteria. These criteria are then routed through a cascade of symbolic rules and encoder/NLI verifiers, which are designed to abstain rather than guess when uncertain. The verdicts from these criteria are aggregated to make a turn-level decision, complete with an evidence trace. SAGE-Core, the recommended operating point, can decide 81-91% of criteria using only the compiler, symbolic rules, and on-device encoders, incurring zero LLM cost. An optional SAGE-LLM fallback can be used for open-class criteria. Across various dialogue datasets, SAGE-Core performs comparably to or better than expensive LLM-as-a-judge baselines, including state-aware GPT-4.1, at a fraction of the cost. Human audits confirm its strong label fidelity, particularly for transcript-visible failure classes.