Global AI routing with <1% overhead on multi-cluster GKE Inference Gateway

2026-09-25

Google's multi-cluster GKE Inference Gateway pools globally distributed accelerator capacity behind a single endpoint, routing LLM traffic based on live KV-cache utilization rather than round-robin. A 17,000-node benchmark across three regions showed near-linear throughput scaling with less than 1 percent routing overhead and a 99.9 percent success rate.