TLDR Tech / Dev
In-House LLM Serving at Netflix
Netflix has developed an in-house LLM serving platform that integrates model deployment and inference within its existing production environment, rather than relying on external hosted APIs. The architecture uses a unified serving system supported by vLLM and Triton, with features like A/B testing and efficient model routing while making smooth transitions from experimentation to production with minimal code changes easy.
Read the full story