Evaluating AI Policy Repair: Aggregate Alignment Can Be Misleading

The study addresses the complex problem of assessing edits made by agentic AI systems to decision policies, particularly when detailed, per-state expert actions are not available for comparison. Researchers used a hotel-pricing simulation where an AI policy editor received only high-level, region-specific diagnostic feedback about its pricing distribution compared to a benchmark. The editor could not access benchmark actions, source code, or direct reward metrics. Results showed that while a multi-restart LLM editor achieved revenue performance very close to the benchmark, a simple diagnostic projection also recovered significant revenue. Crucially, the LLM editor also reduced the "episode composition distance," indicating a more aligned behavior profile beyond just revenue. The research highlights that evaluating agentic policy repair should focus on whether diagnostic feedback reliably translates into desired closed-loop outcomes, rather than just single behavioral distance metrics, as aggregate alignment alone can be deceptive.