Research Probes Memorization in Tabular In-Context Learning Models

Large tabular models (LTMs) leveraging in-context learning (ICL) have achieved state-of-the-art performance on tabular tasks, but their memorization dynamics have remained largely unexplored. This research introduces ICLMEM, a novel probing framework specifically designed to investigate parametric memorization in these models. The framework aims to differentiate between predictions based on contextual patterns and those derived from the model's internal, memorized knowledge. ICLMEM employs a zero-information multiple-choice context to compel the model to rely on its parametric memory, stripping away valid contextual cues. Through a controlled fine-tuning setup, the study establishes ground truth for membership and accounts for common pitfalls like distribution shift and feature contamination. The evaluation on a leading real-world-trained LTM detected moderate memorization signals in 8 out of 10 tasks, with stronger signals observed for low-cardinality and binary tasks. However, these signals were found to largely vanish under more realistic training conditions, suggesting that while a potential risk, it may not be pervasive in typical deployments.