LLMs Struggle with Physics Reasoning in Unfamiliar Worlds

Current benchmarks for Large Language Models (LLMs) in physics often rely on answer accuracy, which doesn't differentiate between genuine reasoning and mere recall of familiar problem types. This research introduces a novel, auditable four-stage diagnostic designed to assess an LLM's ability to reason within unfamiliar physics frameworks. The diagnostic involves induction, formulation, prediction, and review stages, with strict protocols like locked pre-registrations and fresh sessions. The diagnostic was applied to three distinct "parallel physics worlds": a single-equation counterfactual world (F=mv), a historical framework (Aristotelian mechanics), and a four-domain counterfactual world (Decay World). The results for Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro showed low composite PASS rates across these worlds, indicating significant challenges for LLMs in adapting to new physical rules. A key empirical finding was a qualitative-versus-quantitative asymmetry: LLMs rarely predicted the wrong direction of change in Decay World but frequently made incorrect quantitative calculations, often reverting to standard-physics relations. The study also highlighted methodological issues, such as the unreliability of LLM judges across different frameworks and the weakness of LLM self-review, where models often failed to identify their own errors.