LLM Reliability Varies with Prompt Phrasing, Study Finds

New research highlights a significant challenge in Large Language Model (LLM) reliability: their answers can vary dramatically based on subtle changes in prompt phrasing, even when the underlying question remains semantically identical. Despite strong performance on standard benchmarks, which often measure aggregate accuracy, the study found considerable instability at the individual instance level. Across four benchmarks and thirteen different LLMs, the investigation showed that models frequently switch between correct and incorrect answers depending on how a question is worded. Mismatch rates reached over 23%, indicating that a model's ability to answer a question correctly once does not guarantee consistent performance across equivalent inputs. This suggests that standard accuracy metrics can mask substantial underlying instability. Interestingly, the study also observed that models often possess the correct knowledge but fail to retrieve it consistently. Building on this, a simple self-paraphrasing strategy was shown to partially recover this latent knowledge and improve performance during inference. These findings underscore the importance of evaluating LLM reliability beyond mere accuracy, focusing instead on consistency across equivalent inputs to gain a clearer picture of their true capabilities and limitations.