Monarch, Pinterest’s Batch Processing Platform, was initially designed to support Pinterest’s ever-growing number of Apache Spark and MapReduce workloads at scale. During Monarch’s inception in 2016, the most dominant batch processing technology around to build the platform was Apache Hadoop YARN. Now, eight years later, we have made the decision to move off of Apache Hadoop and onto our next generation Kubernetes (K8s) based platform.

Offline batch inference involves operating over a large dataset and passing the data in batches to a ML model which will generate a result for each batch. Offline batch inference jobs generally consist of a series of steps: dataloading, preprocessing, inference, post processing, and result writing. These offline batch inference jobs can be both I/O and compute intensive.

In this blog, we will show how the team transitioned from supporting multiple query serving stacks to provide different data models to a brand new data serving platform with a unified multi model query serving stack called Structured DataStore (SDS).

At Pinterest, we operate a large-scale online machine learning inference system, where feature caching plays a critical role to achieve optimal efficiency. In this blog post, we will discuss our decision to adopt Cachelib project by Meta Open Source (“Cachelib”) and how we have built a high-throughput, flexible feature cache by leveraging and expanding upon the capabilities of Cachelib.

When it comes to PubSub solutions, few have achieved higher degrees of ubiquity, community support, and adoption than Apache Kafka®️, which has become the industry standard for data transportation at large scale. At Pinterest, petabytes of data are transported through PubSub pipelines every day, powering foundational systems such as AI training, content safety and relevance, and real-time ad bidding, bringing inspiration to hundreds of millions of Pinners worldwide. Given the continuous growth in PubSub-dependent use cases and organic data volume, it became paramount that PubSub storage must be scaled to meet growing storage demands while lowering the per-unit cost of storage.

At Pinterest, one of the pillars of the observability stack provides internal engineering teams (our users) the opportunity to monitor their services using metrics data and set up alerting on it. Goku is our in-house time series database that provides cost efficient and low latency storage for metrics data.

Video content has emerged as a favored format for people to discover inspirations at Pinterest. In this blog post, we will outline recent enhancements made to the Adaptive Bitrate (ABR) video performance, as well as its positive impact on user engagement.

In the first part of the post, we introduced the motivations on why we decided to rewrite the ads serving systems and the desired final state. We also outlined our design principles and high level design decisions on how to get there. In part 2 of the post, we are going to concentrate on the detailed design, implementation, and validation process towards the final launch.

Notifications (e.g. email, push, in-app messages) play an important role in driving user retention. In our previous system, which operated on a daily budget allocation model, the system relied on predicting daily budgets for individual users on a daily basis, constraining the flexibility and responsiveness required for dynamic user engagement and content changes. Notification Event Processor (NEP) is a next generation notification system developed at Pinterest, offering the flexibility to process and make decisions to send notifications in near real-time. By harnessing the power of machine learning, NEP determines various factors for sending notifications, such as content selection, recipient targeting, channel preferences, and optimal timing. The implementation of this system resulted in significant improvements in user email and push engagement metrics and weekly active user (WAU) growth.

Pinterest is a visual discovery platform where people can find ideas like recipes, home and style inspiration, and much more. The platform offers its partners shopping capabilities as well as a significant advertising opportunity with 500+ million monthly active users. Advertisers can purchase ads directly on Pinterest or through partnerships with advertising agencies. Due to our huge scale, advertisers get an opportunity to learn about their Pins and their interaction with Pinterest users from the analytical data. This gives advertisers an opportunity to make decisions which will allow their ads to perform better on our platform.

At Pinterest, real-time insights play a critical role in empowering our advertisers and team members to make data-driven decisions. These decisions impact campaign performance, our experiments’ performance, and our policies such as rules to catch spam. We have been using Druid to store and provide these real-time insights, but as our scale and requirements continue to change, we have been evaluating different storage options. In the end we decided to migrate this data to StarRocks.

In this blog post, we’ll discuss and share our experience of launching our Analytics app on StarRocks. In the past, we have published our thoughts on using Druid and the benefits we have gotten from it. This post highlights the need for a new system as our scale and requirements have changed over time.

HBase has been a foundational storage system at Pinterest since its inception in 2013, when it was deployed at a massive scale and supported numerous use cases. However, it started to show significant inadequacy to keep up with the evolving business needs due to various reasons mentioned in the previous blog. As a result, two years ago we started searching for a next-generation storage technology that could replace HBase for many years to come and enable business critical use cases to scale beyond the existing storage limitations.

Pinterest Canvas是一种文本到图像的模型，用于增强Pinterest平台上的现有图像和产品。它通过训练基础文本到图像模型，然后进行微调生成可视化产品的真实背景。模型经过两个阶段的训练，第一阶段训练模型填充缺失的图像区域，第二阶段专注于产品的可视化任务。模型还支持个性化结果，通过附加样式上下文来指导生成过程。Pinterest Canvas的未来改进包括升级底层的扩散骨干模型，进一步提高生成质量，并与用户进行反馈交流。此外，团队还在研究如何重新思考模型条件约束，并探索使用Pinterest优化的视觉嵌入来改进模型的文本条件组件。

Pinterest网站在预发布阶段通过A/B实验和JS捆绑大小检查来主动检测和防止性能回归。当实验的性能指标出现明显回归时，会触发警报和工单，以跟踪和解决问题。此外，通过CI流水线运行的JS捆绑大小检查可以防止捆绑大小增加导致的回归。任何捆绑大小的显著变化都会在PR评论中报告，并发送警报通知相关团队。开发人员可以通过提示解决回归，并使用webpack-bundle-analyzer工具进行进一步调查。

Pinterest在构建Ray基础架构时，采用了中间层解决方案，包括API Server、Ray Cluster / Job Controller和MySQL数据库。该解决方案简化了用户与Kubernetes的交互，并提供了实时的Ray集群和Ray Job状态监控。Pinterest还使用AWS S3持久化Ray日志，并在Ray Cluster UI上展示。他们还开发了Statsboard工具，用于展示Ray应用程序的性能指标和特定功能。此外，Pinterest还提供了三种开发Ray应用程序的选项，并通过ML RESTful API支持Dev server、Jupyter和Spinner工作流。Pinterest还提供了两种测试选项，即Unittest和Integration Test，以及网络和安全性方面的考虑。

Pinterest的广告投放平台是公司最关键的系统之一，负责每年超过30亿美元的收入。为了解决多年的技术债务并为未来的业务目标奠定基础，广告基础设施团队重新设计和重写了该系统。他们设计了一个复杂的软件系统，采用代码组织和团队变更管理等措施，以确保易于扩展、关注点分离、安全设计和开发速度。通过使用内部图执行框架Apex和创新的数据模型，在迁移过程中保证了安全执行。重写后的AdMixer服务已在生产环境稳定运行，实现了产品创新的加速和大团队的安全开发，开发人员满意度得到大幅提升，还实现了基础设施成本的降低。详细设计和迁移过程请参考博文的第二部分。

Pinterest通过监控和调查实时指标来检测和解决Pinner等待时间和核心Web Vital指标的退化问题。他们发现了2023年一些回归问题的根本原因，即对服务器渲染过程进行更改。为了更好地理解网络请求的开始和结束时间，他们引入了网络拥塞时间。这些实时指标帮助他们分析了不同流式传输处理方式对性能的影响，并发现了一些回归问题的关键原因，例如LCP图像的预加载请求延迟、脚本请求开始时间早于LCP预加载请求完成时间等。通过更新日志标记实验和实验处理，他们可以比较实时子指标，进一步提高调查的效率和成功率。这些实时监控图表对于捕捉在生产环境中发布的回归问题非常有帮助。