StarOR: Synergizing Tree Search and Test-Time RL for Optimization Modeling

Optimization modeling, which involves a precise sequence of symbolic decisions, has traditionally relied on methods that are costly to adapt to new problem types or brittle in one-shot generation. Existing search-based approaches often use fixed policies, leading to similar biases and limited credit assignment for intermediate choices. A new framework, StarOR, addresses these challenges by integrating Monte Carlo Tree Search (MCTS) with Test-Time Reinforcement Learning (RL). This synergistic approach decomposes the modeling process into stages, updating a transient LoRA adapter at each non-terminal node using GRPO. StarOR leverages MCTS-generated siblings for local comparisons, transforming exploration into instance-specific policy refinement. It also incorporates an unsupervised multi-faceted reward system to provide fine-grained feedback for intermediate formulation decisions without needing ground-truth labels. This method has demonstrated state-of-the-art performance across various optimization benchmarks, even with a smaller backbone model.