TLDR Tech / Dev
Exploring Speculative Decoding in vLLM on AMD GPUs
Speculative decoding lets vLLM verify multiple draft tokens in one target-model pass while preserving the target model's output behavior. Tests on AMD MI300X and MI355X GPUs show that throughput depends heavily on the drafting method, proposal length, model family, workload, and token acceptance rate, with setup and tuning guidance for MTP, EAGLE-3, DFlash, and DSpark.
Read the full story