KnowSim Evaluates LLM Information Calibration with Learning User Simulators

Large Language Models (LLMs) need to effectively calibrate the information they provide to a user's evolving understanding and cognitive capacity, especially in knowledge-intensive tasks. Current user simulators used for evaluating LLMs often lack explicit models of user knowledge, leading to unrealistic interactions and an inability to track how knowledge changes over time. To address this, a new evaluation framework called KNOWSIM has been introduced. KNOWSIM features a user simulator that maintains explicit knowledge states, represented as a graph of information units with prerequisite relationships. These knowledge states evolve based on learning theory, allowing for more realistic simulations of user interactions. The framework calculates three key metrics: Knowledge Gain, Delivery Calibration, and Cognitive Overload, all derived from the simulated knowledge state trajectory. Validation against 705 human-AI sessions across two domains showed that KNOWSIM's rankings align significantly with human judgments, outperforming existing baseline simulators. When applied to nine different LLMs, KNOWSIM revealed that the best-performing model can vary depending on the user's knowledge level, uncovering "aptitude-treatment interactions" that standard evaluation methods miss. This highlights the importance of nuanced evaluation for LLM assistants.