Repetition Penalty Flaw Corrupts LLM Structured Output

The multiplicative repetition penalty, a common feature across various Large Language Model (LLM) inference ecosystems like HuggingFace and vLLM, contains a critical design flaw. This penalty branches its operation based on the sign of each raw logit, dividing positive logits and multiplying negative ones. However, the softmax function, which converts logits into probabilities, is invariant to adding a constant to all logits. This means a model's logit zero-point is arbitrary and unconstrained by training, yet the sign-branching penalty reads and reacts to this arbitrary point. This fundamental issue leads to two measurable consequences. Firstly, the penalty is not well-defined: re-centering a model's logits, which is a no-op at a penalty factor of 1, drastically changes 58-96% of greedy tokens at a typical factor of 1.3. Different model checkpoints have widely varying zero-points, making a fixed repetition penalty behave differently across models. Secondly, it severely corrupts structured output. On 200 real-world JSON schemas, a penalty factor of 1.3 caused the rate of valid, schema-conformant output to plummet from 97% to just 23%. The research demonstrates that applying the repetition penalty to normalized log-probabilities instead of raw logits effectively removes both of these detrimental effects. HuggingFace already provides a "LogitNormalization" operator, but it is currently off by default and applied after the flawed penalty. The study provides detailed measurements across five models, two code models, and various inference stacks, replicating the effects and confirming the efficacy of the normalized variant.