LLM Agent Skills Decay Silently with Repository Evolution

Large language model (LLM) agents are increasingly used to interact with and operate on software repositories, relying on specific procedural knowledge, or "skills," about APIs, scripts, and conventions. While externalizing this knowledge into reusable skills can boost agent performance, its durability is questionable. These skills are inherently version-specific and can silently become obsolete after a software release, continuing to provide outdated guidance without any explicit signal of decay. The Repo2Skill-Evo study investigates whether agents can maintain the currency of this externalized knowledge. It frames each software release transition as a skill-maintenance task: an agent must update obsolete skill content based on a V1-to-V2 patch while preserving valid guidance. Across 57 real-world repositories and 105 release transitions, every evaluated transition invalidated part of the V1 skill set. However, six frontier agents achieved only 29.9%-69.7% average F1 scores in updating these skills. The dominant errors were either incomplete coverage, leaving stale content untouched, or overbroad editing, which improved recall but reduced precision. This highlights a critical challenge: repository skills decay silently, and even advanced agents struggle to reliably keep them current.