How we optimized Dash's relevance judge with DSPy

2026-03-19

Using DSPy's optimizer, Dropbox defined a clear objective (minimize normalized mean squared error against human judgments), incorporated structured feedback from rating gaps and human rationales, enforced JSON validity guardrails, and ran iterative prompt refinement, enabling 10-100x more synthetic labeling at the same cost, and accelerating model switches from weeks to 1-2 days.