LoKiFormer Boosts LLM Pretraining Efficiency with New Architecture

Large Language Models (LLMs) have made significant advancements, but their pretraining process remains computationally intensive due to architectural inefficiencies. Specifically, standard self-attention mechanisms often redundantly model local information, lacking an explicit bias for locality. Additionally, existing Mixture-of-Experts (MoE) models implicitly link knowledge storage with computational pathways, which limits flexible access to global knowledge. This research introduces LoKiFormer, a new LLM architecture that tackles these issues. LoKiFormer integrates two key modules: Local Fusion Attention (LFA), which uses convolutional fusion to explicitly capture local patterns, allowing attention to focus on more informative representations. The second module is the Knowledge Memory Module (KMM), a parametric key-value memory that stores global knowledge in addressable slots, effectively decoupling knowledge storage from computation and enabling direct retrieval. These innovations allow LoKiFormer to integrate information more efficiently at both local and global levels, leading to faster pretraining convergence.