New Attention Mechanism Enables Exact Unlearning.

Traditional attention mechanisms, while acting as online learners, lack the ability to guarantee that removing a token won't change outputs or to completely erase its influence. This research introduces Support Vector Attention (SV-Attention), a novel memory system built on a max-margin principle, specifically a one-class Support Vector Machine (SVM). This design allows for "certified selection," meaning tokens can be evicted with a guarantee that the model's output remains unchanged. A key innovation is the "exact unlearning" capability. SV-Attention uses a reversible incremental solver that can delete a token's influence precisely, recovering the model state as if it had been trained without that token from the beginning. Experiments confirm that this deletion process is highly accurate, with decision functions matching to a very high degree. While the training process is a batched approximation and doesn't carry the exact-deletion certificate, it still achieves competitive performance. Beyond its theoretical guarantees, SV-Attention demonstrates practical benefits. It significantly improves rare-item recall and reduces deterioration hours on real-world data streams. The system also showcases surgical forgetting, exact editing, patient-record deletion, and a forgettable retrieval memory over real sentence embeddings. In language modeling tasks, a hybrid approach with SV-Attention showed improved performance over standard Transformers.