LOKI Improves Lifelong Knowledge Editing for Language Models

This research introduces LOKI, a novel approach designed to enhance the process of lifelong knowledge editing in large language models. Current methods often struggle with flexibility, as they modify a fixed set of layers for all new information, and require extensive access to previously learned data, which can be a significant hurdle. LOKI addresses these limitations by implementing dynamic layer selection, which adapts based on the specific knowledge being updated. Furthermore, it projects gradient updates into the null-space of the model's weights, a technique that eliminates the need for accessing prior knowledge samples. Experimental results indicate that LOKI achieves superior performance compared to existing methods, demonstrating up to a 14% improvement in average accuracy. This advancement allows for more efficient and less resource-intensive sequential updating of language models while better preserving previously acquired knowledge.