New Index Guides Optimal Data Collection for Few-Shot Learning

A significant challenge in applied machine learning is knowing when to cease collecting additional labeled data. This paper proposes a novel "saturation index," denoted as S(K), which quantifies the ratio of the effective rank of the pooled within-class sample covariance to the number of shots. The authors demonstrate that this index drops below a critical threshold precisely when the covariance estimator becomes stable and the linear discriminant has converged. Crucially, this index can be computed efficiently using only support features, eliminating the need for test labels or a pre-trained classifier. Empirical evaluations across numerous binary classification tasks and datasets reveal a strong positive correlation between the saturation index and marginal accuracy gains. The research identifies a three-phase diagram—exploration, transition, and saturation—each associated with distinct levels of accuracy improvement. The index also proves effective as a binary stopping rule for data annotation, offering probabilistic guidance. Furthermore, a low saturation index combined with poor accuracy can signal representational inadequacy, providing a diagnostic tool for model developers.