提高机器学习模型部署安全标准

ML (machine learning) sits at the heart of how Uber operates at scale. This is made possible by Michelangelo, Uber’s centralized ML platform, which has supported the full ML life cycle since 2016. Today, it runs over 400 active use cases, executes over 20,000 training jobs each month, and serves more than 15 million real-time predictions per second at peak.

机器学习（ML）是Uber大规模运营的核心。这得益于Michelangelo，Uber的集中式ML平台，自2016年以来支持完整的ML生命周期。如今，它运行超过400个活跃用例，每月执行超过20,000个训练作业，并在高峰时每秒提供超过1500万次实时预测。

However, as ML adoption has grown, so have the risks. Unlike traditional code, models are probabilistic and tightly coupled to data—making them harder to validate through static tests alone. A model may perform well offline but fail under real-world conditions due to data drift or integration edge cases. At Uber scale, even small regressions can cause widespread impact within minutes.

然而，随着ML的采用不断增长，风险也随之增加。与传统代码不同，模型是概率性的，并且与数据紧密耦合——这使得仅通过静态测试验证它们变得更加困难。一个模型在离线状态下可能表现良好，但由于数据漂移或集成边缘情况，在现实世界条件下可能会失败。在Uber的规模下，即使是小的回归也可能在几分钟内造成广泛的影响。

In the first half of 2025, we rolled out a series of safety mechanisms aimed at catching issues earlier, validating models more reliably, and mitigating production risk without slowing delivery velocity. This blog explains the thinking behind these mechanisms, how they’re integrated into the ML life cycle (see Figure 1), and how they’re helping Uber raise the bar on model safety at scale.

在2025年上半年，我们推出了一系列安全机制，旨在更早地发现问题，更可靠地验证模型，并在不降低交付速度的情况下减轻生产风险。本文解释了这些机制背后的思考，它们如何融入ML生命周期（见图1），以及它们如何帮助Uber在大规模上提高模型安全性。

Figure 1: Typical ML model life cycle.

图1：典型的ML模型生命周期。

Every model at Uber materializes from two artifacts: data and code. Both are essential—and both are vulnerable to silent failures that may degrade performance or destabilize dependent systems. To design our safe deployment strategy, we started by compiling best practices from MLOps and ModelOps for deploying machine learning models safely. Drawing on industry research and input from ML experts across Uber, we identified a set of safeguards that sp...