Self-CTRL Improves AI Model Transparency and Safety Through Consistency Training.

For AI models, particularly language models, to be truly auditable, understandable, and trustworthy, they must be able to accurately describe their own behavior. The paper introduces Self-Consistency Training with Reinforcement Learning (Self-CTRL), a novel methodology designed to optimize the alignment between a language model's internal explanations and its external actions. Self-CTRL operates by iteratively refining either the model's explanations to better predict its behavior or its behavior to better match its explanations. This dual-direction optimization aims to achieve a high degree of internal consistency. The effectiveness of this method was demonstrated across two distinct domains. In a probabilistic reasoning task, Self-CTRL significantly improved the correlation between a model's self-reported latent biases and its actual behavioral biases. Furthermore, in a constitutional AI setting, the method enabled models to generate rules that faithfully described their compliance or refusal behavior, dramatically increasing the accuracy of third-party auditing. It also substantially reduced the failure rate on harmful prompts while maintaining appropriate responses to harmless ones, thereby making AI models safer, more transparent, and more controllable.