New Metric Explains LLM Prompt Sensitivity

This research delves into the phenomenon of prompt sensitivity in large language models (LLMs), where minor, semantically irrelevant changes to prompts can drastically alter model performance. Traditional evaluation methods often focus on final output changes, which don't explain the underlying causes of this instability. The study proposes a novel approach using "interactions" as a fine-grained tool to analyze prompt sensitivity. By decomposing an LLM's output score into a set of interactions, which represent nonlinear relationships among input variables, the researchers discovered that subtle prompt changes trigger significant instability in these internal interactions, even when the final output appears unchanged. An Interaction-based Prompt Sensitivity (IPS) metric was developed to quantify these interaction changes. Applying IPS to 50 open-source LLMs revealed that supervised fine-tuning, larger model scales, dense architectures, and few-shot learning all contribute to reducing prompt sensitivity. Crucially, these factors primarily achieve this by stabilizing low-order interactions, those involving fewer input variables.