New Attack Recovers Transformer FFN Parameters from Black-Box Access.

This paper unveils a significant security vulnerability in smooth two-layer feed-forward networks (FFNs) commonly found in Transformer architectures. The research introduces a "curvature cryptanalysis" method that allows for the extraction of hidden structural parameters from these FFNs, even under a black-box oracle model where only chosen inputs and raw outputs are accessible, without any knowledge of internal parameters or gradients. The attack exploits a second-order leakage channel: projected input Hessians form distinct mixtures of the same hidden symmetric rank-one factors induced by the FFN input weights. By formalizing Hessian collection as a partially symmetric decomposition, the researchers established conditions for local identifiability and stability. This allowed them to reduce the structural query cost significantly, by a factor of 16, through vector-output stencil reuse. On independently trained CIFAR-10 vision transformers, the method successfully recovered hidden FFN directions with high accuracy using only 16 projected Hessians (8193 black-box queries). This recovery remained robust across different models and blocks. Furthermore, the recovered structure facilitated functional extraction, enabling the creation of high-fidelity substitute models with over 93% top-1 agreement and minimal accuracy loss. While output rounding and Gaussian noise initially reduced recovery, adapting the finite-difference step restored high alignment, demonstrating a robust end-to-end path from black-box observations to internal parameter recovery and functional replacement.