RED-PIM Boosts Transformer Efficiency with Processing-in-Memory

Transformers, widely used across various domains, are often bottlenecked by the extensive data movement between processing units and memory during attention operations. While Processing-In-Memory (PIM) technology offers a solution by performing computations directly within memory, existing PIM-based transformer implementations face challenges with costly inter-bank communication and scalability due to limited memory bank capacity. This often necessitates splitting attention-related data across banks, diminishing PIM's potential benefits. To address these issues, researchers propose RED-PIM, an innovative algorithm-architecture co-design. This framework is specifically engineered to reduce attention latency by drastically minimizing inter-bank data movement from quadratic (O(N^2)) to linear (O(N)) complexity. It also shrinks intermediate attention matrices from N x N to d x d, further optimizing resource usage. RED-PIM achieves these improvements by intelligently reorganizing matrix operations, performing computations locally within memory banks, and employing a highly optimized data transfer strategy. Experimental results demonstrate significant inference time reductions, ranging from 16.05% to 99.99% (geometric mean of 66.42%) compared to baseline PIM implementations, with the most substantial gains observed for longer sequences. On real-world datasets, RED-PIM boosts performance by 99.60% for long documents and 13.44% for shorter ones, all while preserving or enhancing accuracy. This highlights RED-PIM's effectiveness in enabling scalable and efficient transformer inference.