TLDR Tech / AI
How much are AI reasoning gains confounded by expanding the training corpus 10,000x?
Benchmark performance gives biased estimates of out-of-distribution generalization if LLM training data is polluted with benchmark test data. Typical decontamination filters fail to detect semantic duplicates. This suggests that recent benchmark gains are confounded - the prevalence of soft contamination means gains reflect both genuine compatibility improvements and the accumulation of test data and effective test data in the growing training corpora.
Read the full story