TLDR Tech / Practical
Managing LLM-as-a-Judge Systems at Netflix
Netflix engineering authors have outlined a four-stage lifecycle for the development, alignment, and continuous monitoring of LLM-as-a-Judge systems designed to assess recommendation explanations at scale. They introduced Reasoning-Aligned Rubric Tuning (RART), which employs a Meta-Judge to ensure the evaluation model's verdicts are based on sound reasoning, preventing undetected degradation. Live A/B testing confirmed that explanations aligned by the judge successfully influenced member viewing habits toward less familiar content.
Read the full story