New Benchmark Evaluates LLM Parallel Code Translation Reliability

The increasing need to migrate compute-intensive software across diverse hardware accelerators and programming interfaces (like CUDA, OpenMP, OpenCL) has led to interest in using Large Language Models (LLMs) for code translation. However, current evaluation methods often fail to reliably assess whether LLMs preserve the critical low-level parallel semantics required for valid translations, such as thread indexing, synchronization, and memory management. ParBench addresses this gap by providing a kernel-centric benchmark framework for evaluating LLM-based parallel API translation under controlled, executable, and reproducible conditions. It focuses solely on translating computational kernels, fixing the surrounding build and verification infrastructure. The benchmark includes source augmentation to test robustness against surface-form memorization. Evaluations using ParBench on state-of-the-art LLMs expose significant challenges, including directional asymmetry in translation, difficulties with multi-file coordination, incomplete API adaptation, and inconsistent robustness to minor code perturbations.