New AI Model Evaluates Social Influence in Dialogue

Evaluating the effectiveness of social influence dialogue typically relies on surface-level text metrics or single-score LLM judgments, which fail to capture the nuanced changes in a user's internal cognitive states. This research proposes the Cognitive World Model (CogWM), an innovative LLM-based user model designed to assess how a user's beliefs, desires, intentions, and emotions evolve throughout a multi-turn conversation. CogWM functions as both a user simulator and an evaluation platform, employing a three-tier framework that covers turn-level fidelity, trajectory-level state dynamics, and task-level composite scoring. The model was trained using a novel annotation pipeline on over 150,000 user-turn samples across various social influence scenarios. In experiments, CogWM achieved significantly higher emotion accuracy compared to GPT-3.5 and successfully differentiated six commercial agents based on their cognitive influence. This demonstrates CogWM's capability to shift social influence dialogue evaluation from merely judging final outcomes to tracking the intricate process of cognitive state evolution.