Netflix Details LLM-as-a-Judge Lifecycle for Recommendations.

Netflix has unveiled a comprehensive lifecycle framework for its "LLM-as-a-Judge" system, which is instrumental in evaluating the quality of large-scale recommendation explanations. This system processes hundreds of thousands of distinct show-level explanations each week, serving millions of members on their mobile experience. The framework is structured into four critical phases: Birth, Training, Deployment, and Monitoring, emphasizing that an LLM judge is not a static entity but requires continuous management. In the "Birth" phase, evaluation criteria are defined, and human-labeled benchmark datasets are created. "Training" involves refining the judges' rubrics using Reasoning-Aligned Rubric Tuning (RART), a novel procedure that leverages a meta-judge over reasoning output as a learning signal. During "Deployment," the judge serves dual roles: quality gating for new explanations and reflective generation to improve existing ones. Finally, "Monitoring" establishes a continuous Human-in-the-Loop alignment process to detect performance drift and trigger re-tuning. Post-launch A/B tests demonstrated that judge-aligned explanations successfully shifted member viewing towards novel content and increased browse-to-play sessions, validating the system's effectiveness.