TLDR Tech / AI
Keeping Large MoE Training Within Fixed GPU Memory
New scheduling techniques address four key memory bottlenecks in large-scale Mixture-of-Experts (MoE) training: expert dispatch, vocabulary projection, checkpointing, and optimizer state, without approximating computation.
Read the full story