From Weeks to a Day: How We Made LLM Evaluation Fast Enough to Iterate on

2026-07-20

Airbnb cut LLM evaluation cycles from weeks to under a day by caching generated references and judge scores, since over half of model outputs across candidates were identical strings and judge drift of about 1% per run. Small LoRA adapters with rank under 50 train in under an hour on a single GPU for same day fixes, and a final validation stage runs sampled traffic to catch failures.