如何调查DNN模型的在线与离线性能

Predictive model performance gap between offline evaluations and online inference is a common and persistent challenge in the ML industry, often preventing models from achieving their full business potential. At DoorDash, this issue is particularly critical for deep learning models, as it impacts multiple teams across domains. Bridging this gap is essential for maximizing the business value of these models.

离线评估与在线推理之间的预测模型性能差距是机器学习行业中常见且持续的挑战，常常阻止模型实现其全部商业潜力。在DoorDash，这个问题对深度学习模型尤其关键，因为它影响多个领域的团队。弥合这一差距对于最大化这些模型的商业价值至关重要。

The Ads Quality ML team encountered the same challenge for our latest few ranking model iterations. In this blog, using the latest launched model iterations as a case study, we will walk through the debugging process, and share a scalable methodology framework for investigating and resolving these discrepancies. 

广告质量机器学习团队在我们最近的几次排名模型迭代中遇到了相同的挑战。在这篇博客中，我们将以最新发布的模型迭代作为案例研究，逐步介绍调试过程，并分享一个可扩展的方法论框架，以调查和解决这些差异。 

By adopting the solution proposed in the blog, we reduce the online-offline AUC gap from 4.3% to 0.76%.

通过采用博客中提出的解决方案，我们将在线与离线AUC的差距从4.3%减少到0.76%。

Our experience highlights critical areas such as feature serving consistency, feature freshness, and potential concerns when integrating real-time features for model serving. These insights can guide future efforts to improve offline and online performance alignment.

我们的经验强调了关键领域，例如特征服务一致性、特征新鲜度，以及在集成实时特征进行模型服务时可能出现的担忧。这些见解可以指导未来的努力，以改善离线和在线性能的对齐。

Restaurant Discovery Ads, the primary entry point for ads in the app, contributes the largest share of ad revenue. Key milestones since early 2023 are summarized in Fig-1. After evaluating various model architectures, we have adopted the Multi-Task Multi-Label (MTML) model architectures.

餐厅发现广告是应用程序中广告的主要入口，贡献了最大的广告收入。自2023年初以来的关键里程碑在图1中总结。经过评估各种模型架构，我们采用了多任务多标签（MTML）模型架构。

Figure 1: Restaurant Discovery Ads Ranking Deep Learning Milestones

图1：餐厅发现广告排名深度学习里程碑

The goal for this milestone (M4), Multi-MTML V4 is to add more features to further improve the model performance. For the online-offline AUC gap in...