Netflix 内部的 LLM 服务

By AI Platform’s Model Runtime team and Inference team

由AI Platform的Model Runtime团队和Inference团队撰写

Introduction

简介

Most organizations consume LLMs through hosted APIs. Netflix went further — we run the full stack ourselves, from model deployment through inference, inside our existing production environment rather than a separate ML silo. Some of those decisions weren’t obvious, and a few revealed their trade-offs only under production load.

大多数组织通过托管 API 消费 LLM。Netflix 走得更远——我们在现有的生产环境内部，而不是在独立的 ML 孤岛中，自己运行从模型部署到推理的整个技术栈。其中一些决定并不显而易见，而且有些只有在生产负载下才显现出它们的权衡。

This post focuses on the choices where alternatives were seriously considered: engine selection, model packaging, API surface design, deployment strategy, and output constraints enforcement. The goal is to share not just what was built, but why — and what production revealed that the design phase didn’t anticipate.

本文重点介绍那些经过认真考虑替代方案的决策：引擎选择、模型打包、API 表面设计、部署策略以及输出约束执行。目标不仅是分享构建了什么，还有为什么这样构建——以及生产环境揭示了哪些设计阶段未曾预料到的问题。

Architecture Overview

架构概述

Member-scale ML at Netflix is fronted by a unified JVM-based serving system that handles the end-to-end flow for downstream consumers: routing and A/B test logic, candidate generation, feature fetching, inference, post-processing, and logging at each stage. Both real-time and cached batch paths are supported. Figure 1 shows the two ways callers reach inference today: the gRPC path through this serving system and a direct HTTP path used by newer LLM-driven applications.

Netflix 的会员级机器学习由一个统一的基于 JVM 的服务系统作为前端，该系统为下游消费者处理端到端流程：路由和 A/B 测试逻辑、候选生成、特征获取、推理、后处理以及每个阶段的日志记录。支持实时和缓存批处理路径。图 1 展示了当前调用方进行推理的两种方式：通过该服务系统的 gRPC 路径，以及较新的 LLM 驱动应用程序使用的直接 HTTP 路径。

Where inference runs depends on the model. Small CPU models run in-process, avoiding remote-call overhead. Larger models need GPUs — the serving system handles pre- and post-processing locally but delegates inference to a remote service, Model Scoring Service (MSS). MSS is the shared inference backend, supporting XGBoost, TensorFlow, PyTorch, and LLMs behind a unified interface, with N...