TLDR Tech / AI
A Controlled Study of Attention-Only Transformers
The feed-forward network (FFN) has been a fixed component of the transformer block since its introduction. In modern decoder configurations, it holds roughly two-thirds of non-embedding parameters. A substantial interpretability literature argues that these layers act as the model's parametric memory. This experiment looks at what is actually lost when the FFN is removed entirely.
Read the full story