KANs Explored for Small Language Models: No Clear Advantage

Kolmogorov-Arnold Networks (KANs) have been proposed as a potentially more interpretable and efficient alternative to traditional feed-forward networks (MLPs) in neural architectures, replacing fixed activation functions with learned one-dimensional edge functions. This research investigates the claims of KANs, particularly their suitability for small language models (SLMs). The study aimed to determine if KANs could offer superior performance or interpretability compared to standard MLPs. The findings indicate that while KANs with a small basis can indeed provide a practical interface for auditing the scalar transformations learned by the network, this interpretability does not consistently translate into performance benefits. Experiments on various benchmarks, including BabyLM and Wikitext-103, showed that KAN-family variants, despite sometimes improving validation loss over GELU MLPs, did not consistently outperform strong MLP baselines in terms of accuracy or latency on standardized tasks. Specifically, larger KAN models underperformed established baselines like SwiGLU. The research suggests that properties like high compression and closed-form fit are more characteristic of low-capacity KAN bases rather than universal KAN behavior. Therefore, while KANs offer a novel approach to network design and interpretability, they do not yet present a clear, consistent advantage over well-optimized MLP architectures for language models.