Netflix Details Its In-House LLM Serving Platform with Triton and vLLM

Netflix has developed an in-house platform for serving Large Language Models (LLMs), which it calls Triton. This infrastructure is designed to handle the complexities of supporting different model sizes, hardware requirements, and rapidly evolving inference engines. The company’s account provides a detailed look at the production lessons learned from building this platform.

The platform builds on Netflix’s existing JVM-based serving layer, which continues to handle tasks such as routing, feature retrieval, candidate generation, post-processing, and logging. Smaller models can run in-process on CPUs, while larger requests are delegated to MSS, where Triton takes over model loading, batching, GPU scheduling, and multi-framework serving.

This approach allows the surrounding production workflow to remain consistent even as inference moves between local and remote hardware. Within the GPU path, Netflix selected vLLM for its operational fit and extensibility while retaining Triton’s model-management and scheduling responsibilities. Triton controls the serving environment around the model, whereas vLLM performs inference and provides extension mechanisms for custom behavior.

Netflix reports that mismatched Triton and vLLM versions can prevent deployments from loading, requiring compatible releases to be tested and pinned together. Custom models introduced another integration challenge. Hugging Face compatibility in vLLM was insufficient for some Netflix models, so the company used vLLM extension points for custom architectures and decoding behavior.

Netflix also compared two Triton packaging approaches: Triton’s Python backend and its vLLM backend. The company reports that the vLLM-backend approach allows models and frontends to evolve more independently than the Python-backend option. This choice affects how tightly a model is coupled to its serving environment rather than which engine performs inference.

The common serving interface did not eliminate differences between the underlying engines. Although Triton exposed an OpenAI-compatible API alongside KServe HTTP and gRPC frontends, Netflix still encountered gaps in how some features were handled across those integrations. Constrained decoding was one example of this issue.

Constrained decoding allows Netflix to force model responses into formats such as valid JSON by filtering the tokens the model may generate at each step. Because these rules depend on everything generated so far, the decoder must maintain state throughout the request. When vLLM pauses and later resumes a request to manage GPU resources, that state can fall out of sync with the token history.

Netflix added logic to detect this change and rebuild it before generation continues. Compatibility also affected deployment. Netflix pins tested Triton and vLLM versions together to prevent backend-loading failures, while Red-Black and Versioned deployment strategies handle changes at the model level. Versioned deployments keep old and new revisions available separately, allowing consumers to migrate after adapting to incompatible input or output schemas.

Uber has described a related approach at the application boundary. Its generative AI gateway presents an OpenAI-compatible interface across externally hosted and internally managed models, while centralising concerns including authentication, caching, observability, and routing. The implementation differs from Netflix’s serving platform, but both separate application integrations from the models, runtimes, and hosting environments behind them.

Netflix’s experience shows how a common serving interface can sit above several distinct layers. This type of architecture reflects a broader effort to give application teams a stable integration surface while model providers and serving runtimes continue to change. Netflix’s account also shows that the abstraction does not remove the underlying work: packaging, compatibility controls, constrained decoding, and deployment isolation still require engineering at each layer.