TLDR Tech / AI
Popping the GPU Bubble
AI models typically produce one token at a time - you can't compute the third token before you have the second. The GPU does most of the heavy lifting, but there is also some work that needs to be done by the CPU. GPU bubbles occur when the GPU sits idle in a loop waiting for the GPU to complete its job. This article looks at how to hide these bubbles using a technique called pipeline decoding, which involves starting the GPU work on the next token while the CPU is still finishing the last one.
Read the full story