TLDR Tech / AI
Mixture-of-Depths Attention (GitHub Repo)
MoDA introduces a new attention mechanism that lets each head access both current-layer and earlier-layer key-value pairs, helping preserve useful signals as models scale deeper.
Read the full story