Tabular Foundation Models Show Internal Inconsistencies.

A recent study delves into the internal consistency of Tabular Foundation Models (TFMs), which are transformer-based models designed to approximate Bayesian posterior predictive distributions for tabular data. While TFMs are considered leading approaches for tabular prediction, their ability to generate coherent multivariate predictions by autoregressively sampling targets and adding them to features had not been thoroughly examined. The researchers posed a fundamental question: can a TFM's predictions genuinely arise from a single, underlying joint distribution? To answer this, they established two critical requirements: marginalization consistency (marginalized conditionals must equal directly predicted marginals) and factorization consistency (different factorization orders must yield identical joint distributions). Across various datasets and for both classification and regression tasks, every TFM evaluated was found to violate both of these consistency requirements. This suggests a significant internal inconsistency, implying that TFMs do not produce predictions derivable from a single, faithful joint distribution.