TLDR Tech / Practical
Marin 535B-A23B: Model Design and Training Insights
This article explores the design choices, rationale, and initial training results for Marin 535B-A23B, a substantial mixture-of-experts model trained on 18 trillion tokens. It discusses architectural decisions influenced by GB200 system hardware efficiency, the pursuit of stability, and risk mitigation strategies employing scaling ladders.
Read the full story