Hierarchical Solomonoff Induction Offers Unbounded ML Sequence Prediction

Solomonoff Induction (SolInd) is recognized as an ideal, unbounded model for a priori sequence prediction. However, it struggles to naturally account for extrapolation from a given training dataset, a core function of modern Large Language Models. To bridge this gap, researchers have developed Hierarchical Solomonoff Induction (HSI). HSI applies de Finetti's theorem on exchangeable distributions to SolInd, creating a model that maintains a hyperprior over all Solomonoff priors. This structure enables HSI to be conditioned on previously observed sequences, making it suitable for learning from datasets. The research proves that HSI is equivalent to SolInd, demonstrating that universal mixtures of mixtures are also equivalent to SolInd. Furthermore, HSI's excess error on any distribution, when compared to its true generator, is bounded by the generator's complexity within the hyperprior. This result is analogous to SolInd's prediction error being bounded by the Kolmogorov complexity of the sequence. Crucially, HSI's average excess error converges to zero as a dataset grows, establishing it as an optimal model for sequence prediction given a dataset, mirroring SolInd's optimality for individual sequences.