Can Language Models Independently Discover the Concept of Zero?

A fundamental question in the development of AI systems, particularly those based on artificial neural networks, is their capacity for mathematical discovery and their ability to generalize beyond their training data. This involves hypothesizing genuinely new, and potentially more powerful, mathematical structures. Human cognition suggests that language abilities might support such generalizations. This study uses simple arithmetic as a case study to explore how modern AI models might expand their mathematical understanding, specifically examining if they can independently discover the concept of "zero." The research reveals two key findings. First, language models of a GPT-2 size are unable to perform this generalization at test time, regardless of whether they underwent language pretraining. This indicates a limitation in their ability to spontaneously infer a novel mathematical concept without direct exposure. Second, the models' performance substantially improves after being trained on even a relatively small number of examples (tens or hundreds) involving the concept of zero. Furthermore, the study found that prior language pretraining significantly reduces the number of required examples by approximately 50%. This suggests that while language models may not discover "zero" entirely on their own, their existing language abilities can act as a scaffold, making the process of mathematical discovery more efficient once some initial examples are provided.