Guided Protein Language Models Suffer Off-Manifold Collapse

Protein language models are increasingly used for designing new protein sequences, often guided at inference time to optimize specific properties. However, this guidance faces a dilemma: strong guidance, while effective at moving the desired property, can lead to generated sequences that are difficult or impossible to fold. This research identifies a specific failure signature called "off-manifold collapse," where the model's internal representations degenerate to a state statistically similar to random amino acid input. Crucially, the property oracle being optimized often fails to detect this collapse, sometimes even rewarding these biologically implausible sequences. To address this, the researchers propose Mahalanobis filtering, a training-free, post-hoc step. This method applies a cheap density prior over natural protein activations, keeping only candidates that remain typical under this prior. This filtering significantly improves both the property score and the structural plausibility of the kept sequences at negligible cost, without modifying the generator itself, and is transferable across different guidance methods.