TLDR Tech / Dev
The Benchmark-pocalypse
LLM agents have made it effortless to "reward hack" and overfit performance benchmarks, allowing developers and startups to boast impressive speedups that do not translate to real-world performance. An experiment where an unsupervised AI agent optimized a regex engine to beat benchmark suites without true architectural improvements showed that software benchmarks are increasingly untrustworthy without proper auditing.
Read the full story