Detecting Misalignment in LLM Agent Skills with Contrastive Learning

Large Language Model (LLM) agents are increasingly enhanced through "Agent Skills," which are reusable components packaging natural-language metadata, procedural instructions, and execution resources. As open-source skill marketplaces grow, users and agents rely heavily on brief metadata to select third-party skills. This reliance creates a significant challenge: detecting inconsistencies between a skill's advertised description and its actual operational behavior, a problem termed "cross-layer misalignment." To tackle this, researchers propose Progressive Loading-Aware Hierarchical Contrastive Learning (PL-HCL), an LLM-based framework. PL-HCL is designed to model the layered structure of Agent Skills and learn consistency across these layers. By understanding how different parts of a skill (metadata, instructions, resources) relate, the framework can identify discrepancies that indicate misalignment. Evaluated on a corpus of over 264,000 open-source skills and a human-verified challenge set, PL-HCL demonstrated substantial improvements. It boosted Macro-F1 scores from approximately 0.45 for unadapted baselines to 0.87-0.89 across various LLM backbones. This approach provides an effective screening tool for users and operators, and offers valuable design principles for ensuring consistency in layered digital artifacts, enhancing trust and reliability in agent skill marketplaces.