OraclePhys Framework Improves LLM Fine-Tuning for Structural Mechanics

A new framework named OraclePhys has been developed to systematically fine-tune large language models (LLMs) for structural mechanics tasks. This framework comprises three core components: OraclePhys-Bench, a benchmark that provides exact, oracle-scored evaluations for structural mechanics problems without human labeling; OraclePhys-30K, a supervision dataset offering seven distinct answer forms for byte-identical structure descriptions; and a controlled study examining how different answer forms and verifier roles impact training. The study yielded two significant findings. Firstly, the specific format of the answer label, rather than its data size, critically dictates what the fine-tuning process imparts to the LLM. For instance, a ranking objective can instill an out-of-distribution forward model, while scalar objectives offer only partial understanding, and boolean objectives show no detectable learning. This distinction persists across different physics domains and model families. Secondly, the research indicates that explicitly written or score-filtered answers are effective in building this capability, whereas advantage-weighted scores (like GRPO) improve reward but fail to statistically differentiate the model from its initial state on unseen physics problems. The fine-tuned 8B LLM, the first of its kind for spatial structural response, achieved performance at the data-precision frontier, surpassing other frontier LLMs and matching specialist levels.