TLDR Tech / Practical
AWS primes models to alter architecture without full retraining
Researchers at Amazon Web Services have developed priming, a technique that repurposes pretrained Transformer weights to drastically cut the cost of pretraining hybrid state space models by over 99%. By substituting certain attention layers with state space layers like Gated KalmaNet, the method overcomes Transformer memory limitations. The researchers found these hybrid models can achieve comparable reasoning benchmarks to Transformers while decoding at twice the speed. To encourage further development, the team has released primed hybrid models, training code, and a vLLM serving plugin under the Apache 2.0 License.
Read the full story