TLDR Tech / AI
Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter
Prefill-as-a-Service (PrfaaS) is a cross-datacenter serving architecture that selectively offloads long-context prefill to standalone, compute-dense prefill clusters and transfers the resulting KVCache over commodity Ethernet to local PD clusters for decode. It combines model-side KV efficiency with system-side selective offloading, bandwidth-aware scheduling, and cache-aware request placement. The design removes the requirement that heterogeneous accelerators share the same low-latency RDMA fabric and enables independent scaling of prefill and decode capacity across loosely coupled clusters. A PrfaaS-augmented heterogeneous deployment achieves higher serving throughput while consuming only modest cross-datacenter bandwidth.
Read the full story