Lean Proof Assistant Enhances Reinforcement Learning for Theorem Proving

Reinforcement learning from verifiable rewards (RLVR) typically relies on a simple binary signal to indicate success or failure. However, symbolic proof assistants like Lean offer a much richer, structured form of feedback. This research bridges the gap between structured processes and unstructured rewards by demonstrating how the Lean proof assistant can serve as a symbolic process oracle, providing both overall outcome and fine-grained tactic-level verified feedback during the training of theorem-proving agents. Proof attempts are analyzed by parsing them into sequences of tactics. Lean's elaboration process then marks each step as locally sound or identifies the earliest failing step. This provides dense, verifier-grounded credit signals rooted in type theory, offering a more detailed understanding of the proof process than a simple pass/fail. These structured rewards are integrated into a GRPO-style reinforcement learning objective, incorporating first-error propagation and first-token credit methods. Experiments with existing theorem provers like STP-Lean and DeepSeek-Prover-V1.5 show that this tactic-level supervision consistently outperforms baselines that only use outcome-level feedback, leading to improvements on benchmarks such as MiniF2F and ProofNet. The study highlights that symbolic proof assistants can function not just as evaluators but as crucial process-level reward oracles during training, paving the way for more reliable and scalable RL frameworks in formal reasoning.