Microsoft Develops AI Agent Regression Eval Set Curation Pipeline

Platform teams that host agent-extensibility surfaces face a significant challenge: while every new customer brings their own domain-specific evaluation set, the platform's overall regression set must operate within strict query-count limits due to release cadences. There's a recognized gap in published industrial solutions for this platform-side curation problem, as existing frameworks are typically customer-focused, and benchmark compression research doesn't account for a continuous stream of incoming evaluation sets. Microsoft 365 Copilot has developed a novel, capability-taxonomy-driven curation pipeline specifically for declarative agents with custom actions. This pipeline takes an agent specification and a customer's evaluation set, projects each query into a platform-owned capability taxonomy, and then decides whether to admit, drop, swap, or flag a query for human review. The core philosophy is to maintain a regression set that is minimal yet captures the broadest possible spread of capability signatures—unique combinations of capabilities exercised by a query. The pipeline consists of three main components: a classifier that uses both deterministic specification-based extraction and large language model (LLM) semantic inference to determine per-query capabilities; an Invocation Quality (IQ) rater that scores how thoroughly a query exercises each capability, allowing better tests to displace existing ones; and a consolidator that compares incoming queries against the regression set based on coverage and quality through a rule-based decision cascade. This mechanism is designed to be taxonomy-agnostic, making it applicable to any regression evaluation-set curation problem with a typed capability taxonomy, even those that evolve over time.