Google Proposes Random Masking for LLM Optimization

2026-02-19

Google researchers showed that randomly masking parameter updates can outperform dense adaptive optimizers, with a masked RMSProp variant exceeding recent state-of-the-art methods. Their proposed Magma optimizer uses momentum-aligned gradient masking to improve pre-training efficiency.