LLM Explanations Often Insufficient, Research Finds

As large language models are increasingly used in critical applications, the reliability of their free-text explanations, such as chain-of-thought, becomes paramount. This paper investigates whether these explanations truly provide enough information to understand the model's output-generating process. Researchers propose a novel information-theoretic metric called SCSuff (Self-Consistent Sufficiency) to evaluate explanation sufficiency. This metric leverages the LLM itself to generate alternative inputs conditioned on an explanation, effectively capturing the model's internal beliefs about possible inputs. Experiments reveal that LLM explanations are often insufficient, meaning they don't fully capture the model's reasoning. The sufficiency also depends on the input data distribution and shows little correlation with model size, accuracy, or output entropy. The study suggests that internal representations can predict SCSuff scores, potentially guiding improvements in explanation quality.