New Method Reduces LLM Sycophancy in Autism Intervention Dialogues

Large language models, when used in sensitive applications like clinical autism intervention, can exhibit sycophancy, where they agree with users even if incorrect, posing safety risks. While standard fine-tuning helps, it often falls short in precisely correcting these failure patterns. This new research introduces a technique called TD-DPO, or Token-level Difference Direct Preference Optimization, designed to specifically target and reduce sycophantic behaviors. The core innovation involves a "Minimal Edit Data Augmentation" (MEDA) strategy. This creates highly controlled preference pairs by making only tiny, critical edits between a preferred and a rejected response. TD-DPO then leverages these pairs by giving more weight to the tokens that differ between the chosen and rejected responses, while downplaying shared tokens. This prevents the model from over-updating parts of the response that are not relevant to the sycophancy issue, thereby preserving its overall intervention capabilities. Extensive testing across various model architectures and evaluation methods demonstrated that TD-DPO effectively balances sycophancy reduction with the retention of the model's primary intervention functions. This suggests a promising new approach for aligning AI models in high-stakes clinical settings, particularly where precise behavioral control is paramount.