Transformers Learn Number Theory Heuristic for Elliptic Curve Rank Prediction

A recent study explored the capabilities of transformer models in number theory, specifically in classifying rational elliptic curves. A two-layer transformer encoder was trained to predict whether an elliptic curve has rank 0 or rank 1, achieving high accuracy exceeding 99%. Through detailed mechanistic interpretability techniques, including attention analysis and neuron-level circuit analysis, the researchers uncovered the underlying algorithm learned by the model. They found that a sparse circuit of specific neurons was sufficient for accurate prediction. Crucially, the model's learned input weights for the top discriminating neuron closely matched the Mestre-Nagao sum heuristic weights, a known result from analytic number theory. This indicates that the transformer independently discovered a complex mathematical principle from raw Frobenius trace data.