处理 Pinterest 广告排名系统中的在线-离线差异

Author: Cathy Qian, Aayush Mudgal, Yinrui Li and Jinfeng Zhuang

作者：Cathy Qian，Aayush Mudgal，Yinrui Li和Jinfeng Zhuang

Image from https://unsplash.com/photos/w7ZyuGYNpRQ

图片来自https://unsplash.com/photos/w7ZyuGYNpRQ

Introduction

介绍

At Pinterest, our mission is to bring everyone the inspiration to create a life they love. People often come to Pinterest when they are considering what to do or buy next. Understanding this evolving user journey while balancing across multiple objectives is crucial to bring the best experience to Pinterest users and is supported by multiple recommendation models, with each providing real-time inference with an overall latency of 200–300 milliseconds. In particular, our machine learning powered ads ranking systems are trying to understand users’ engagement and conversion intent and promote the right ads to the right user at the right time. Our engineers are constantly discovering new algorithms and new signals to improve the performance of our machine learning models. A typical development cycle involves offline model training to realize offline model metric gains and then online A/B experiments to quantify online metric movements. However, it is not uncommon that offline metric gains do not translate into online business metric wins. In this blog, we will focus on some online and offline discrepancies and development cycle learnings we have observed in Pinterest ads conversion models, as well as some of the key platform investments Pinterest has made to minimize such discrepancies.

在 Pinterest，我们的使命是为每个人带来创造自己喜爱生活的灵感。人们经常在考虑下一步要做什么或购买什么时来到 Pinterest。理解这种不断变化的用户旅程，并在多个目标之间取得平衡，对于为 Pinterest 用户带来最佳体验至关重要，这得益于多个推荐模型的支持，每个模型都提供实时推断，整体延迟为 200-300 毫秒。特别是，我们的机器学习驱动的广告排名系统正在努力理解用户的参与和转化意图，并在适当的时间向适当的用户推广正确的广告。我们的工程师不断发现新的算法和新的信号，以提高我们的机器学习模型的性能。典型的开发周期包括离线模型训练以实现离线模型指标的提升，然后进行在线 A/B 实验以量化在线指标的变化。然而，离线指标的提升未必能转化为在线业务指标的胜利。在本博客中，我们将重点关注 Pinterest 广告转化模型中观察到的一些在线和离线差异以及开发周期的经验教训，以及 Pinterest 为减少此类差异所做的一些关键平台投资。

What, Why, and How

什么、为什么和如何

During our machine learning model iteration, we usually imple...