Eval-driven development: Lessons from evaluating GenAI at scale

2026-07-30

Airbnb evaluates GenAI by reviewing 100 prototype outputs to find real failure modes before creating automated evals. Its stack uses programmatic checks, calibrated LLM judges with agreement in the high 80% range against human reviewers. Agentic systems are assessed across tool calls, reasoning steps, and final outputs.