SkillConsist Detects Inconsistencies in LLM Agent Skills

Large Language Model (LLM) agents rely on "skills" for reusable capabilities. However, inconsistencies between what a skill declares it does and what it actually implements can lead to dangerous behaviors or incorrect skill selection. Current methods for detecting these inconsistencies often struggle with the mixed nature of text and code in skill definitions and the complex mapping between declarations and multiple implementation steps.SkillConsist tackles these issues by first using an LLM to separate declaration and implementation content into distinct behavior records, supplemented by static analysis for implementation. These records then form two separate graphs. The core of SkillConsist is a bidirectional graph alignment process that searches for corresponding subgraphs between the declaration and implementation sides.Once aligned, a graph differencing mechanism identifies conflicts, pinpointing inconsistencies. Tested on a benchmark of 633 skills, SkillConsist significantly outperforms existing baselines in both package-level detection and localization of inconsistencies, demonstrating its effectiveness in ensuring the reliability and safety of LLM agent skills.