LLM Oversight Flawed: Correct Answers Can Hide Faulty Reasoning

The practice of using chain-of-thought monitoring for AI oversight often involves providing human or AI monitors with a trusted reference answer. A recent study investigated whether access to this answer genuinely improves the verification of an AI's reasoning or merely helps identify incorrect final conclusions. Researchers analyzed 237 step-numbered solutions from three frontier LLMs to physics questions, identifying both final answer correctness and the first false step. They found 24 critical instances where the final answer was correct despite an error in the reasoning trace. Eight LLM monitors evaluated these traces under different conditions: blind, with an unverified or certified answer, or after a blind commitment. The study concluded that certified answer access significantly boosted the detection of wrong-answer traces but paradoxically reduced the flagging of errors in critical traces (where the answer was correct but the reasoning flawed). This suggests that answer access primarily aids in checking consistency with the conclusion rather than independently verifying the argument's soundness. This phenomenon, akin to reward hacking, highlights a critical safety concern where acceptable outputs can mask unsound underlying processes.