TLDR Tech / Data
How an AI Token Travels Through a Data Center
Inference has become the dominant AI cost center: it now consumes roughly two-thirds of AI compute and 80–90% of lifetime model cost. The economics hinge on cost per token at target latency, with prefill being compute-bound and decode memory-bandwidth-bound, making KV cache management, batching, quantization, and speculative decoding the main levers. Durable advantage sits in physical bottlenecks: HBM bandwidth, NVLink/scale-up fabric, optics, and power.
Read the full story