Auditing Reveals Flaws in LLM Self-Improvement Measurement

The assessment of language model improvement often focuses on individual problem gains and losses rather than just overall accuracy, which can be susceptible to measurement artifacts. This study rigorously audited three rounds of rank-32 LoRA self-training on Qwen3-8B, comparing it against an identical pipeline with a frozen control model. The audit uncovered seven significant measurement failures, each capable of reversing reported findings if the control group was not present. Several of these failures are common practices, such as using a single greedy decode to track capability changes, which can manufacture improvements in an untrained model due to inference batching. The study found that standard threshold repair methods also failed replication. To address these issues, the researchers propose a per-problem exact test against a pooled baseline, controlled for false-discovery rate. This new method detected no improvements on held-out replicates for self-training. Applying this robust auditing framework to various training arms, the study concluded that external distillation effectively improves problems the base model struggles with. In contrast, three forms of self-training did not show similar gains and, in fact, corrupted problems that the base model could already solve at rates above a measured floor. This highlights the critical need for a separately measured null for every reported statistic in transition-level auditing, ideally derived from baseline replicates already available in multi-arm studies.