Tabular Foundation Models Boost Discrete Choice Estimation Accuracy

Tabular Foundation Models (TFMs) offer a promising approach for predictions on structured data through in-context learning, bypassing the need for task-specific estimation. However, directly applying TFMs to discrete choice problems, a fundamental framework in marketing and operations for demand estimation, yields limited performance. This performance gap arises because TFMs typically assume row-independent observations, whereas discrete choice inherently involves set-valued observations and significant consumer preference heterogeneity. To overcome these structural limitations, researchers propose a reformulation that effectively encodes both choice-set dependence and individual heterogeneity within a row-based learning framework. This novel approach was evaluated using a yogurt scanner panel dataset. The study found that encoding individual-level heterogeneity was the primary driver of predictive accuracy. The most effective reformulation significantly outperformed traditional hierarchical Bayesian estimation, achieving an 8% improvement in holdout log-likelihood and a 3.6% increase in hit rate. Crucially, it also ran 16 times faster, offering a substantial practical advantage for large-scale demand estimation. The benefits were particularly pronounced in the medium-data regime (10-40 purchase occasions per consumer), where parametric Bayesian shrinkage often distorts estimates for atypical consumers. Fine-tuning the model on population choice data further improved gains for consumers with limited purchase histories, where in-context learning alone might lack sufficient individual-specific signals. These results establish a robust method for applying foundation models to a broader range of consumer choice problems.