TLDR Tech / Practical
Tessera Enhances Data Mixing for AI Pretraining
A new automated method called Tessera has been developed to improve data mixing for language and genome model pretraining. It generates thousands of data sources based on individual document performance during training. Rather than using manually created topic categories, Tessera groups documents by their impact on training results and identifies optimal mixtures across numerous possibilities. Tests show these derived sources offer about four times the improvement in held-out loss compared to optimized topic mixtures, while allowing for precise control over performance.
Read the full story