CurveFP Datatypes Optimize Low-Precision Language Model Training and Inference

Reducing the cost of language models often involves using low-precision datatypes, but many existing formats prioritize scalar fidelity without optimizing the arithmetic operations, particularly products. CurveFP addresses this by introducing a new family of closed-product codebook datatypes. These datatypes distribute quantized magnitudes across interleaved logarithmic curves, managed under compact block scales. A key innovation is the use of a rational radix, which allows for tuning dynamic range against local resolution. Uniform curve indices ensure that every non-zero product is algebraically closed, simplifying product formation to an exact sign XOR and integer-index update. A derived finite phase count then dictates the accumulation schedule. The research instantiates this algebra as CurveFP eight (E4C3/E5C2) for training and CurveFP seven (E3C3) for compact deployment. Evaluations show that CurveFP seven surpasses tensor-wise FP8 perplexity on several 7B-9B models, using one fewer element bit while staying within 1.32% of native quality. CurveFP eight also demonstrates lower operand NMSE across numerous GEMM comparisons. In pretraining experiments, CurveFP eight achieved mean BF16-inference perplexity comparable to FP8, and its checkpoints consistently showed lower WikiText-103 perplexity in downstream tasks. These results establish CurveFP as an arithmetic co-design that combines FP8-class numerical behavior with seven-bit inference and a significantly simpler product path, offering substantial efficiency gains for language models.