Dual Teachers Improve Adversarial Robustness and Accuracy.

Deep neural networks (DNNs) are highly susceptible to adversarial attacks, where small, imperceptible perturbations to input data can cause misclassifications. Existing countermeasures, such as Information Bottleneck Distillation (IBD), often improve robustness but at the cost of reduced accuracy on clean, unperturbed inputs. Researchers have proposed an enhancement to the IBD framework to mitigate this trade-off. Their method introduces a dual-teacher approach, incorporating an additional "clean teacher" model trained exclusively on clean data, alongside the traditional robust teacher trained with adversarial examples. Features from both teachers are then transferred to a student model via a cross-layer attention mechanism. Experimental results on standard datasets like CIFAR-10 and CIFAR-100 demonstrate that this dual-teacher strategy significantly improves classification accuracy on clean samples compared to the original IBD, while preserving similar levels of robustness against adversarial attacks. The approach also proves competitive with other state-of-the-art distillation frameworks, particularly in balancing clean and robust accuracy.