TLDR Tech / DevOps
Building an AI factory on Kubernetes
Building an AI factory on Kubernetes means solving two problems at once: keeping GPU utilization high and keeping tenants from interfering with each other on the same hardware. The stack to do it now exists mostly from CNCF and open source projects, including DRA (GA in Kubernetes 1.34), HAMi for software-based GPU partitioning, vCluster for per-tenant virtual control planes, KServe for inference, and OpenCost for per-tenant chargeback. Network and storage isolation increasingly get pushed into hardware, with DPUs such as NVIDIA BlueField handling enforcement off the host CPU.
Read the full story