Agent Skill Representation Impacts Multimodal AI Retrieval Performance

This paper presents a case study from Tinycloud, a production multimodal video agent harness, examining how the representation of skills influences an LLM planner's ability to discover and select the most appropriate tool for a user's task. At smaller scales, this selection often occurs directly within the LLM's context window, without explicit embedding-based retrieval. The study categorizes skills into "tool-skills" (wrapping single APIs) and "workflow-skills" (orchestrating multiple tool-skills), exposing them in the system prompt via either full inlined instructions or a concise one-line listing. An ablation study across different exposure regimes demonstrated that full autoloading consistently selected the correct skill. However, partial exposure, the default production setting, led to misrouting in one task due to lexical competition where an autoloaded tool-skill diverted the planner's attention from a more suitable listed workflow-skill. The key finding is that simply exposing more skills in the prompt is not always beneficial; partial exposure can introduce conflicts that hinder accurate skill selection. This observation has implications for larger-scale retrieval-based skill routing, suggesting that careful consideration of skill representation and exposure is crucial for robust agent performance.