Boundary-Seeking Distillation Fails in Bottlenecked Generative AI

Data-free knowledge distillation is a technique for transferring knowledge from a large "teacher" AI model to a smaller "student" model without needing the original training data. For classifiers, methods like Contrastive Abductive Knowledge Extraction (CAKE) achieve this by synthesizing samples near the teacher's decision boundary. This study investigated whether this boundary-seeking principle could be extended to autoencoder distillation, specifically using the MNIST dataset. To facilitate a direct comparison, the researchers reframed continuous reconstruction as a dense, per-feature classification task, allowing the decoder to output categorical logits. Their findings reveal that boundary-seeking objectives are inherently ill-suited for bottlenecked generative architectures. Unlike classifiers, where CAKE operates on a single, instance-level objective, a decoder functions as an array of tightly coupled, feature-level classifiers, all constrained by a shared, low-dimensional bottleneck. The core problem is that independently sampling contrastive targets for these interconnected outputs disrupts the geometry of the learned latent manifold. This leads to significant gradient conflicts rather than generating informative boundary samples. The research concludes that a manifold-aware synthesis approach is necessary to bypass these conflicts, establishing a more effective baseline for data-free generative distillation.