A Decision-Only Judge Matches GPT-6 on Routine Evals for 0.36% of the Fee

2026-09-25

CMU tested Jev, a judge that returns only a verdict and label probabilities, against 16 LLM and reward-model judges. It comes within three points of GPT-6 on preference and factuality at 277 times lower fees, though it fails on difficult correctness tasks.