使用请求级去重扩展推荐系统

At Pinterest, scaling our recommendation models delivers outsized impact on the quality of the content we serve to users. Our Foundation Model (oral spotlight, ACM RecSys 2025), for example, achieved a 100x increase in transformer dense parameter counts and a 10x increase in model dimension; translating directly into meaningful quality improvements across multiple recommendation surfaces.¹

在 Pinterest，扩展我们的推荐模型对我们为用户提供的内容质量产生了超额影响。例如，我们的 Foundation Model (oral spotlight, ACM RecSys 2025) 实现了 transformer dense 参数计数的 100 倍增加和模型维度的 10 倍增加；这直接转化为多个推荐表面的有意义的质量改进。¹

But a 100x scaleup creates massive infrastructure pressure. Storage, training, and serving costs all threaten to grow proportionally unless you’re deliberate about efficiency. The single highest-impact technique we’ve deployed to hold costs in check across all three dimensions is request-level deduplication: a family of techniques that ensures we process and store request-level data once, not once per item.

但是 100 倍扩展会产生巨大的基础设施压力。除非你对效率有意识，否则存储、训练和服务成本都会成比例增长。我们部署的单一最高影响技术，用于在所有三个维度控制成本，就是 request-level deduplication：一组技术，确保我们只处理和存储一次请求级数据，而不是每个项目一次。

In this post, we’ll walk through what request-level deduplication is, why it matters so much for modern recommendation systems, and how we applied it across the full ML lifecycle , from storage compression to training correctness and speedups to serving throughput gains.

在本文中，我们将介绍 request-level deduplication 是什么，为什么它对现代推荐系统如此重要，以及我们如何将其应用于整个 ML lifecycle，从存储压缩到训练正确性和加速，再到 serving throughput 提升。

Background

背景

A request is triggered when a user opens their fe...