OEIS Benchmarks Primarily Measure Memorization, Not Induction

Integer sequences from the On-Line Encyclopedia of Integer Sequences (OEIS) are widely used to evaluate the mathematical reasoning abilities of large language models. However, new research questions what these benchmarks truly assess. By employing a computable reference learner based on two-part Minimum Description Length (MDL) for P-recursive recurrences, researchers made several key findings. First, MDL difficulty correlates with parameter count, with the "discovery point" (when a symbolic hypothesis beats verbatim storage) being predictable by combinatorial bounds. This point is invariant to term magnitude, indicating that the underlying structure, not the numbers themselves, dictates difficulty. Second, the study found a "wilderness" regime where models acquire a theory for a sequence prefix but then lose it at full length, suggesting induction often fails to generalize. Third, evaluating three language models against these MDL regimes showed that models don't confabulate in the "wilderness" but rather hedge appropriately. Surprisingly, confident errors were concentrated on "easy" sequences, implying that apparent competence often stems from sequence recognition rather than true rule induction. This suggests OEIS benchmarks largely measure memorization, and MDL provides a valuable, contamination-free difficulty signal.