REFORGE Benchmarks LLM Reverse Engineering Capabilities in Binary Naming

This research introduces REFORGE, a meticulously designed pipeline for benchmarking the reverse engineering capabilities of Large Language Models (LLMs), particularly in the task of naming functions within decompiled binaries. The authors highlight a significant flaw in existing benchmarks: they often assume perfect ground truth for function-level evaluation, overlooking the complexities introduced by compiler optimizations. This oversight can lead to inaccurate assessments of LLM performance. REFORGE tackles this by creating provenance-tracked ground truth, moving from C source code through compilation, DWARF and syntactic extraction, alignment, and decompilation. Crucially, it operationalizes alignment uncertainty using an eight-gate confidence funnel and three-tier stratification. A micro-benchmark demonstrated that high-confidence ground truth yield drops significantly with optimization levels, and traditional unpaired comparisons can misrepresent performance decay due to survivorship bias. A proof-of-concept evaluation of seven LLMs on function naming underscores the necessity of this uncertainty-aware benchmarking practice for fair and reliable assessment.