LLM-Generated Skills Show No Improvement for Data Science Workflows.

The study explores the utility of Large Language Model (LLM)-generated "skill files" in enhancing the performance of LLM-based agents assisting with data science tasks. Product data scientists often use LLMs for recurring tasks like data cleaning, SQL writing, statistical test selection, and result formatting. Skill files are designed to provide reusable guidance, avoiding the need to prompt from scratch for similar task families. While expert-written skills offer high-quality guidance, their creation and maintenance can be a bottleneck. The researchers questioned whether LLM-generated skills could serve as a low-curation alternative, specifically if they improve performance over simply prompting with the task alone. They tested this across four data science lifecycle stages: data preparation, extraction, statistical analysis, and reporting, using one generated skill per stage. The findings indicate no reliable improvement from using full generated skills compared to "No-Skill" prompting. A comprehensive ablation study, involving 56 tasks, nine model configurations, and three providers (totaling 7,560 runs), further confirmed that neither the full generated skill nor any ablated component significantly improved performance. All p-values were high, and the total performance spread across variants was minimal. A token-matched control experiment also showed similar performance for full skills and task-irrelevant content. These results caution against relying on LLM-generated skills as a default single-shot prompting strategy for data science workflows.