New Method Transfers Knowledge Across AI Models of Different Sizes

A novel technique called Activation-Prune-Merge (APM) has been introduced to facilitate knowledge transfer between large and small language models. This method allows a smaller model to gain capabilities from a more powerful, larger model without requiring extensive retraining or precise structural alignment between the two. The core idea involves identifying and extracting the most functionally relevant parts of the larger model based on its activation patterns. APM works by creating task-specific activation maps on the larger "donor" model. It then intelligently prunes specific layers, hidden dimensions, attention heads, and MLP neurons to match the architecture of the smaller "recipient" model. This "donor slice" is then injected into the recipient model using a small interpolation coefficient, treating the donor as a source of concentrated functional components rather than requiring a full structural transplant. Experiments across 16 benchmarks, including reasoning, math, and code generation, showed that APM improved the average accuracy of a 3B recipient model from 55.5% to 60.6%. Specific tasks like RTE saw accuracy jump from 64.3% to 82.3%. This demonstrates that effective cross-scale heterogeneous fusion is possible without explicit semantic alignment, provided the donor's contribution is carefully selected and concentrated.