TLDR Tech / AI
Scaling Data Repetition for LLMs
The optimal amount of high-quality domain data repetition increased mildly with model size at a fixed tokens-per-parameter ratio. Smaller proxy models could therefore help estimate repetition schedules for larger models, with lower-loss domains generally tolerating more reuse.
Read the full story