New Method Improves Verifiable Explanations for LLM Activations

Researchers have identified a significant flaw in how explanations for Large Language Model (LLM) hidden activations are typically evaluated. Existing natural-language autoencoders, which score explanations based on their ability to reconstruct the original activation, often fail to penalize individual false claims. This means an explanation can achieve a high reconstruction score even if it contains inaccuracies, as long as the overall "gist" is preserved. The study found that such methods can lead to models developing "private codes" or false wording that the reconstruction depends on, without reflecting true internal states. To address this, the paper introduces RECAP (Readable Encodings via Co-trained Auxiliary Predictors), a novel approach that trains linear heads alongside the target LLM. RECAP's goal is to make specific internal content independently decodable and verifiable. This method ensures that designated information within the model becomes reliably accessible to external probes, rather than relying solely on the model's own prose explanations which can be misleading. Experiments show that RECAP significantly improves the truthfulness of explanations and enhances robustness against adversaries attempting to manipulate reconstruction scores while lying.