CASTER Enables Gradient-Free Test-Time Adaptation for Frozen Models

A new research paper presents CASTER, a novel gradient-free approach to Test-Time Adaptation (TTA) designed for scenarios where model parameters cannot be updated during inference. This method is particularly beneficial for deployments on inference-only accelerators, with frozen or third-party models, or in memory-constrained environments, overcoming limitations of traditional gradient-based TTA. CASTER operates by storing source class statistics within a discriminative subspace and then estimating a class-shared affine transformation from the moments of the target batch. It analytically transports these source class distributions before classification, requiring no backward pass, optimizer state, or stored source feature bank. The method consistently outperforms k-NN on identical frozen features across various datasets and backbones, while maintaining significantly less state. The paper also introduces an empirical residual-to-margin transportability certificate to assess the reliability of the affine transport, especially in challenging scenarios like ImageNet-C where unconditional transport can lead to significant performance drops. This certificate helps gate the adaptation process, converting potential performance losses into gains and providing a safety signal for deployment.