TLDR Tech / Data
PagedAttention: Virtual Memory for the KV Cache
PagedAttention brings virtual-memory-style paging to LLM KV caches by replacing contiguous per-request allocation with fixed-size blocks and a block table. That design eliminates most cache fragmentation, raises utilization to roughly 96%, supports copy-on-write for shared prompts, and can serve 2–4x more users per GPU without changing attention outputs.
Read the full story