Wiola: A Novel Architecture for Efficient Small Language Models

Researchers have unveiled Wiola, an entirely new Small Language Model (SLM) architecture that breaks away from existing model families like GPT or LLaMA. This innovative design is built from fundamental principles, incorporating five distinct and novel components aimed at enhancing efficiency and performance. Key innovations include Spiral Rotary Positional Encoding (SRPE) for richer positional signals, Gated Cross-Layer Attention (GCLA) for improved inter-layer coherence, and Adaptive Token Merging (ATM) to dynamically reduce attention complexity. Additionally, Wiola features a Dual Stream Feed-Forward (DSFF) network and a modified normalization technique called WiolaRMSNorm to prevent representation collapse. The architecture is thoroughly detailed with mathematical derivations and complexity analyses, demonstrating its unique approach. Wiola models are released in various sizes (120M to 1.5B parameters) and are fully integrated with the HuggingFace Transformers ecosystem, ensuring ease of adoption and testing for developers.