TabPATE Enables Private Tabular In-Context Learning Without Public Data

This research introduces TabPATE, a novel method designed to enhance privacy in tabular in-context learning (ICL). The core problem addressed is the vulnerability of private records used in ICL to leakage through model predictions, even with basic inference attacks. TabPATE offers a differentially private solution, leveraging a PATE-style defense mechanism.The technique works by distributing the private contextual data among multiple "teacher" models. These teachers then privately aggregate their labels on synthetically generated tabular queries. The resulting labeled queries are subsequently released as a student context, which is safe to use. This approach is particularly effective because tabular features are often bounded and low-dimensional, allowing for useful queries to be generated from feature ranges or lightly privatized marginals.Evaluations across various tabular benchmarks demonstrate that TabPATE maintains competitive utility while significantly reducing the success rate of membership inference attacks to near-random levels. This provides a practical pathway for implementing private tabular ICL, especially in scenarios where access to public in-distribution data is not feasible.