TLDR Tech / AI
Introducing Attention Residuals: Rethinking depth-wise aggregation
Residual connections have long relied on fixed uniform accumulation. Attention Residuals replace standard depth-wise recurrence with learned, input-dependent attention over preceding layers. It enables networks to selectively retrieve past representations, naturally mitigating dilution and hidden-state growth. Attention Residuals have been validated on the Kimi Linear architecture, demonstrating consistent downstream performance gains.
Read the full story