Lyft的全谱系ML模型监测

Photo by Vasundhara Srinivas on Unsplash

Machine Learning models at Lyft make millions of high stakes decisions per day from physical safety classification to fraud detection to real-time price optimization. Since these ML model based actions impact the real world experiences of our riders and drivers as well as Lyft’s top and bottom line, it is critical to prevent models from degrading in performance and alert on malfunctions.

Lyft的机器学习模型每天做出数百万的高风险决策，从物理安全分类到欺诈检测到实时价格优化。由于这些基于ML模型的行动影响到我们的乘客和司机的现实体验以及Lyft的盈亏，因此防止模型的性能下降和对故障发出警报是至关重要的。

However, identifying and preventing model problems is hard. Unlike problems in deterministic systems whose errors are easier to spot, models’ performance tends to gradually decrease, which is more difficult to detect. Model problems stem from diverse root-causes including:

然而，识别和预防模型问题是困难的。与确定性系统中的错误更容易被发现不同，模型的性能倾向于逐渐下降，这更难被发现。模型问题源于不同的根源，包括：。

Bugs in the caller service which passes wrong features or incorrect units to the model (garbage in results in garbage out)
调用者服务中的错误，将错误的特征或不正确的单位传递给模型（乱入导致乱出）。
Unexpected change in an upstream feature definition
上游特征定义中的意外变化
Distribution changes of input features (Covariate Shift)
输入特征的分布变化（协变量转移）。
Distribution changes of output labels (Label Shift)
输出标签的分布变化（标签转移）。
Conditional distribution changes for output given an input (Concept Drift)
给定输入的输出的条件分布变化（概念漂移）。

One example of a model problem happened when Estimated Times of Arrival (ETAs) fell in response to declined demand due to COVID. The ETA models were retrained since they were over-predicting ride times due to being trained on historical demand. This, however, caused other models which took ETAs as input to dramatically under-predict on pricing, revealing one of the many challenges we hadn’t originally anticipated.

模型问题的一个例子是，由于COVID导致的需求下降，估计到达时间（ETAs）下降。ETA模型被重新训练，因为它们是根据历史需求训练出来的，所以过度预测了乘车时间。然而，这导致其他以ETA为输入的模型对定价的预测大大不足，揭示了我们最初没有预料到的许多挑战之一。

In early 2020, motivated by examples like t...