TLDR Tech / DevOps
Why goodput matters more than throughput for LLM serving
A benchmark of the LLM serving framework vLLM on a single NVIDIA A10G GPU revealed that maximizing throughput can degrade user experience, with one configuration achieving 50% higher token throughput but causing token streaming delays nearly 10 times worse (494ms vs 50ms at p95). The study demonstrated that "goodput"—requests meeting both throughput and latency targets for time-to-first-token and time-per-output-token—is a more meaningful metric than raw throughput alone, with the optimal configuration varying significantly across chatbot, reasoning, and agentic workloads.
Read the full story