Context Sampling Impacts TabPFN Accuracy and Stability

This research delves into the impact of context sampling choices on TabPFN's performance for classification on small tabular datasets. TabPFN operates via in-context learning, where it conditions on a set of labeled training rows (context) to make predictions without gradient updates. The study investigated how context size and selection methods affect prediction stability, accuracy, and selection cost across 15 OpenML datasets. Key findings indicate that increasing context size substantially boosts both accuracy and stability, with the AUC coefficient of variation decreasing significantly for larger contexts. While accuracy correlates with distribution representativeness in random contexts, controlled experiments showed that merely matching feature means could reduce accuracy by compromising context diversity. Mixed-effects analysis confirmed that diversity and coverage are stronger predictors of accuracy than feature-mean matching. Furthermore, expensive selection methods like K-Means and farthest-point sampling provided no accuracy benefits over uniform random sampling, despite incurring significantly higher costs. This suggests random sampling is effective due to its expected feature-space coverage.