TLDR Tech / Practical
ReplaySSM Enhances Hybrid-Model Decoding with Smarter State Storage
A new technique called ReplaySSM has been developed to improve the efficiency of state-space model decoding. It stores the model's state as a combination of a checkpoint and a buffer for recent inputs, which helps alleviate memory bandwidth limitations. This method speeds up decoding and speculative verification, and uses different precisions to avoid errors from quantization. ReplaySSM has been integrated into vLLM and TensorRT-LLM, boosting performance for hybrid models like Nemotron 3 and Qwen3.5.
Read the full story