TLDR Gripe / Practical
LLM Parallelism Explained: Data, Tensor, Pipeline, and Expert Methods
This article explores the integration of data, tensor, pipeline, and expert parallelism in large language models. It presents a practical example of deploying a 35-billion-parameter mixture-of-experts model across eight GPUs. The study shows how expert parallelism modifies memory distribution and GEMM configurations by leveraging tensor and data parallelism groups, without affecting the total expert weights on each GPU. Furthermore, the author details how combining expert parallelism with DP-attention minimizes KV cache redundancy and allows for larger batch sizes, thus improving overall performance.
Read the full story