Notes on pretraining parallelisms and failed training runs

2026-05-18

Pretraining runs often fail. This article looks at all the ways that things can go wrong and why training is such a precarious operation. The key culprits seem to be breaking causality and adding bias.