TLDR Tech / Dev
How Google made their Gemma LLM 3x faster
Google's Gemma model achieved a three-fold increase in inference speed using speculative decoding. This technique involves a tiny, fast 'Drafter' predicting tokens that the main model verifies in parallel to efficiently use the GPU and save memory.
Read the full story