TLDR Tech / Dev
What Actually Keeps an AI Benchmark Useful? Scale
A study of 60 widely used LLM benchmarks shows that nearly half have reached saturation, making them unable to differentiate top models due to measurement noise. Some factors contributing to saturation include benchmark age and test set size, while commonly assumed safeguards like private test sets and output formats do not effectively prevent this issue.
Read the full story