TLDR Tech / Data
Pretraining progress is mostly coming from data
A controlled comparison of 2019-2025 open-model recipes and data corpora attributes 12.0x compute-efficiency gains to data improvements versus 3.7x to model recipes at a 1e19-FLOP budget. The result frames dataset extraction, curation, and filtering as a major systems lever, while cautioning that larger-scale and synthetic-data effects remain untested.
Read the full story