HASTE System Boosts ML Engineering Efficiency with Skill Transfer.

This paper introduces HASTE (Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineering), a multi-agent system designed to prevent ML engineering agents from repeatedly "rediscovering" solutions in new competitions. The core idea is to efficiently organize and transfer knowledge by structuring it into three distinct scope tiers: global, domain-specific, and competition-specific. An orchestrator agent manages these tiers, coordinating domain specialists and facilitating knowledge abstraction and promotion between levels using Large Language Models (LLMs). A controlled experiment demonstrated the significant benefits of this scoped knowledge loading. With a fixed inventory of 159 skills across 8 competitions, HASTE achieved a 100% medal rate, whereas a flat loading approach (without hierarchical organization) only reached 62.5%, performing no better than loading no skills at all, and consumed twice the computational tokens. On the comprehensive MLE-Bench Lite benchmark, which includes 22 Kaggle competitions, HASTE achieved a 77.3% medal rate using Claude Sonnet 4.6, with each competition taking approximately 12 hours. The system showed remarkable efficiency gains in "warm-start" runs, where it leveraged previously learned global and domain-level skills. Warm starts required 52% fewer refinement iterations, and the fraction of proposed changes accepted by the agent dramatically increased from 42% with low skill inventory to 85% once over 50 skills were available. These findings suggest that intelligent knowledge organization can effectively substitute for raw model strength and computational budget in ML engineering tasks.