We Tested Jev on 100 Real Agent Calls. How Easy Is It to Beat a Constant?

2026-09-22

Tests on 100 production agent tool calls found that 79% were benign, so a classifier had to beat a constant benign baseline before its accuracy was useful. Jev's errors, class imbalance, and benchmark design show why evaluations need deployment-aware baselines and why a strong-looking score can still lose to simple code.