SkillEval Interprets and Improves LLM Agent Skill Quality

As AI agents increasingly rely on reusable "skills" or procedural knowledge to tackle specialized tasks, accurately evaluating the quality of these skills becomes critical. Current evaluation methods often assess skill quality by measuring performance on specific downstream tasks. However, this approach provides only a partial view, as a skill might apply to many scenarios, and it doesn't pinpoint which aspects of the skill need improvement. Researchers have developed SkillEval, an interpretable framework designed for document-level skill evaluation. SkillEval identifies general properties within a skill's documentation (e.g., `SKILL.md`) that are crucial for its quality. It evaluates each property using a fixed and inspectable scoring direction, generating interpretable scores that are less influenced by superficial document features like length or formatting. Specifically, SkillEval learns these interpretable directions in the hidden representation space of a model, using controlled positive-negative skill pairs. It then scores new skills by projecting their representations onto these directions. Experiments show that SkillEval reliably differentiates skills of varying quality, and its scores correlate well with downstream task performance, offering an early indicator of a skill's utility. Furthermore, SkillEval can diagnose weaknesses in skill documents and guide targeted revisions, leading to improved skill properties and higher success rates in agent tasks.