Understanding LLM Agent Skills: When and Why They Fail

Skills have emerged as a popular method to enhance Large Language Model (LLM) agents by providing structured knowledge packages at inference time. However, most evaluations focus on aggregate task success, leaving a gap in understanding the specific conditions under which skills help, why they are effective, and where they ultimately fail. Through controlled experiments across various benchmarks, agent harnesses, and LLMs, researchers isolated the effects of skill representation, outcome annotation, retrieval difficulty, and cross-framework robustness. A contrastive study, combining quantitative experiments with trajectory analysis, revealed that skills primarily function as "procedural anchors," stabilizing agent execution rather than injecting new facts. This procedural anchoring accounted for 65.7% of skill cases, compared to only 4.5% for explicit knowledge injection. The study also identified retrieval as a critical bottleneck; as skill pools expanded from 5 to 100, actual-use precision plummeted from 29.6% to 3.3%. Skills are prone to failure when assumptions are brittle, contexts are incompatible, or adaptation is insufficient. These findings move beyond simple success rates to provide a deeper understanding of skill mechanics and limitations for building more reliable self-evolving agents.