TLDR Tech / Data
In-House LLM Serving at Netflix
Netflix runs LLM serving end-to-end in-house, using a unified JVM-based platform with gRPC plus an OpenAI-compatible HTTP API, rather than relying on hosted APIs. The stack standardizes deployment through Triton-backed Model Scoring Service, using vLLM as the paved-path engine after re-benchmarking. It also adds custom packaging, zero-downtime rollouts, and patched guided decoding to prevent malformed JSON output.
Read the full story