安全高效的多轮嵌入训练

Authors: Shruthi Hariharan | Machine Learning Engineer II; Konish Dutta | Machine Learning Engineer II; Abhinav Naikawadi | Sr. Machine Learning Engineer; Zhihua Zhang | Machine Learning Engineer II; Rakshita Nagalla | Sr. Machine Learning Engineer; Kousik Rajesh* | Machine Learning Engineer II; Matt Lawhon | Staff Machine Learning Engineer
Tags: Machine Learning, Infrastructure, Recommendation Systems
Content Track: Technical Discussion / Architecture of big components

* == work done while at Pinterest

* == 在 Pinterest 任职期间完成的工作

Introduction

引言

Large ID embedding tables underpin some of the largest modeling advances at Pinterest, including teacher-model scaling and cross-domain pretraining for ranking and retrieval. These tables are first pre-trained to learn broadly useful Pin representations and then fine-tuned as part of the downstream surface-specific ranking model. Here, we’ll focus on the Homefeed ranking model.

大型 ID 嵌入表支撑了 Pinterest 一些最大的建模进展，包括教师模型扩展和面向排序与检索的跨域预训练。这些表首先经过预训练，以学习广泛有用的 Pin 表征，然后作为下游特定场景排序模型的一部分进行微调。这里，我们将重点放在首页信息流排序模型上。

The strength of the embedding tables comes from their enormous capacity to absorb information about billions of Pins and learn rich representations. That power, however, is coupled with fragile training dynamics that have historically constrained fine-tuning to one epoch.

嵌入表的强大之处来自其巨大的容量，能够吸收关于数十亿个 Pins 的信息并学习丰富的表示。然而，这种能力伴随着脆弱的训练动态，历史上一直将微调限制在一个 epoch 内。

That compromise has become increasingly dissatisfactory; single-epoch fine-tuning prevents us from effectively optimizing our models over small, fresh datasets in retraining, and disproportionately harms effective representation learning of certain components of our ranking model which benefit from longer training, like Vision Language Models (VLMs). A one-epoch ceiling forces every com...