LLMs Exhibit Ideological Drift in News-Grounded QA

Large language models (LLMs) are increasingly used to process and answer questions about political information, a domain where factual accuracy and neutrality are paramount, especially during elections. A new measurement framework has been developed to diagnose ideological drift in LLMs by treating hallucinations—statements unsupported by the source document—as key signals.The study utilized over 21,000 expert-labeled U.S. political news articles from various ideological sources. Researchers generated article-specific questions, elicited answers from several LLMs (both open-weight and proprietary), and then identified sentence-level hallucinations. These hallucinated sentences were subsequently classified for their ideological valence using a fine-tuned stance classifier.Findings indicate that while the frequency of hallucinations varies across models and concentrates in contentious topics, the content of these hallucinations consistently exhibits a robust leftward drift. A majority of hallucinated sentences were classified as left-leaning, even when the LLM was processing right-leaning source material. Logit-level analysis suggests that hallucinations often arise in high-uncertainty generation contexts, and for some models, this uncertainty also correlates with leftward drift, implying an "uncertainty to guessing" mechanism. This research has significant implications for auditing AI-mediated political information and designing safeguards.