TLDR Tech / InfoSec
Why agents that pass tests fail in production
Mercor co-founder Adarsh Hiremath argues that agents that pass one-shot tests still fail in production because success is measured by output rather than trajectory, and that production-grade eval suites need continuous, representative monitoring rather than one-time validation. He recommends a staged rollout, a personal assistant, sandboxed tools, real users without real data, then phased production, to limit the blast radius, while 1Password's Nancy Wang frames non-human identity governance as the harder, unresolved problem, since agents authenticate via long-lived tokens that don't generate the audit trails security teams rely on.
Read the full story