New Benchmark for Adversarial Plan-Generation Agent Evaluation

A new benchmark, AdvPlan-Bench, has been developed to evaluate structured plan-generation agents under adversarial conditions. Unlike traditional evaluations that assess plan quality in isolation, this benchmark considers how a plan performs when an opposing agent actively seeks optimal responses. The core contribution is a general evaluation object that includes a typed plan, an adversarial response set, selector diagnostics, and traceable candidate-frontier metrics. AdvPlan-Bench represents plans as typed action chains with optional branches, assigning synthetic quality scores and comparing opposing plans using BLUE-vs-RED advantage and Nash-gap diagnostics. It also evaluates qualitative constraint coherence with a transparent heuristic rubric. The benchmark was tested across 150 synthetic scenarios, demonstrating that a sampled best-response policy significantly reduces the advantage and win rate of a given plan compared to single-sample responses. The study also included baselines, showing that an offline LLM-policy contract achieved a certain advantage, while a two-stage multi-agent council performed better. A rubric-sensitivity study confirmed high inter-rater agreement. It's important to note that AdvPlan-Bench is a reproducible artifact for studying adversarial plan evaluation and not an operational planner, providing no direct evidence about real-world decision quality.