TLDR Tech / AI
Parallelizing Transformer Training
This interactive guide explores data parallelism, FSDP, tensor parallelism, pipeline parallelism, and expert parallelism for training transformers. It focuses on how different hardware and communication patterns determine when each strategy becomes bottlenecked.
Read the full story