Personalized LLM Judges Enhance Recommendation System Evaluation

Traditional methods for evaluating recommendation systems offline often rely on complex, manually maintained feature pipelines that struggle with scalability. While large language models (LLMs) offer a potential alternative by predicting user engagement from raw text, initial studies reveal a critical flaw: "bidirectional rationalization." This phenomenon shows that off-the-shelf LLMs can convincingly argue for contradictory user engagement outcomes using the same evidence, making them unreliable for direct evaluation. To overcome this, researchers developed a sequential behavioral alignment framework. This framework combines fine-tuning with preference optimization, utilizing paired correct and counterfactual rationales. When applied to real-world homepage interaction logs, this aligned reasoning method achieved a substantial improvement in Macro-F1 score compared to zero-shot LLM baselines, matching the performance of established production feature-engineered baselines. The results demonstrate that this behavioral alignment effectively addresses the issue of bidirectional rationalization while providing human-interpretable reasoning without the overhead of manual pipeline maintenance.