Hierarchical Grading Enhances Large Language Models

The paper introduces Graded Large Language Models (GLLMs), an innovative algebraic framework that enhances the representation space of transformer models with a "grading." This grading is systematically applied across embeddings, self-attention mechanisms, and the training objective, extending the theory of graded neural networks to autoregressive language models. Crucially, this enhancement preserves the expressive power, asymptotic computational complexity, and inference cost of standard transformers. The underlying mathematical foundation for GLLMs is geometric invariant theory. This framework enables the identification of optimal grades—specific configurations that offer improvements over uniform architectures. These optimal grades can be determined through a convex program prior to training, based on measurable profiles of the target data, effectively certifying the optimal grades before any training commences. For tasks involving level-stratified targets, the research demonstrates a significant separation in risk between graded and uniform target classes, showing that GLLMs can achieve superior performance within an explicit window of sample sizes. After training, the grading is absorbed into the learned parameters, meaning a GLLM compiles into a standard transformer with identical architecture and inference complexity, offering performance benefits without added operational overhead.