For over a decade, we've accepted that end-to-end backprop is the only way to train deep networks

2026-05-29

Holding the entire network in memory at once is why AI training is hitting a resource wall. Sakana Labs has found a new way to break the network into blocks and train them independently. The trick was to treat the network's forward pass like a diffusion model denoising a signal. This slashes the memory needed to train deep models.