LLM Explanatory Engagement Varies with Anomaly Rarity and Elicitation.

A study explored how the explanatory behavior of large language models (LLMs) shifts when they encounter increasingly rare anomalous failures within a structured workflow. Previous research focused on whether LLMs detect anomalies; this work narrows the scope to how their "explanatory engagement"—measured by length, specificity, and self-reported confidence—changes once an anomaly is recognized and its failure rate diminishes. Using three open-weight models (Qwen3:8b, Llama3.1:8b, Mistral:7b) in a repeated tool-call task with varying failure probabilities, the researchers initially hypothesized a rise in engagement followed by a collapse. While a pooled analysis showed a monotonic decline in explanation length, splitting the data by elicitation condition revealed a more nuanced picture. Under immediate and forced explanation conditions, the predicted rise in engagement was observed, with explanation length peaking at moderate rarity and confidence generally increasing. This suggests that how and when an LLM is prompted to explain failures critically influences its response, highlighting elicitation structure as a primary moderator of observable behavior.