KANs Show Promise for Small LLMs, But No Clear Advantage Yet

Researchers investigated Kolmogorov-Arnold Networks (KANs) as a potential replacement for standard feed-forward networks within small language models. KANs are designed with learned one-dimensional edge functions, which theoretically provide a more explicit and interpretable interface for understanding how models process information. The study focused on auditing these networks and evaluating their performance as replacements. The audit revealed that a significant portion of KAN's feed-forward edges are active and can be pruned with minimal performance loss, similar to structured MLP pruning. However, the claimed benefits of near-total compression and high closed-form fit coverage were found to be specific to low-capacity KAN configurations, not a universal property. When tested as replacements for MLPs, KAN-family variants did show improvements in validation loss on BabyLM datasets. Yet, these gains did not translate consistently to standardized benchmarks like BLiMP, where KANs showed no clear or consistent advantage over strong MLP or SwiGLU baselines, and even underperformed in larger tests. The findings suggest KANs offer a practical interface for auditing learned transformations but lack a consistent performance edge over established architectures.