从自研到 Flink：大规模迁移有状态广告事件 Join

Background

背景

We are the Ad Platform team at Zalando Marketing Services (ZMS). We build the data pipelines that power Sponsored Products, the paid placements that advertisers use to promote their items across Zalando's catalog and product card pages.

我们是 Zalando Marketing Services (ZMS) 的 Ad Platform 团队。我们构建的数据管道为 Sponsored Products 提供支持，Sponsored Products 是广告商用于在 Zalando 的目录和产品卡片页面推广其商品的付费展示位。

When a customer sees or clicks a sponsored item, that interaction needs to be matched against the original auction that placed it. The auction record carries the ad server's delivery decision context; the interaction event carries the result. Joining the two is what turns a raw click into a billable charge and closes the feedback loop to our ad serving system.

当客户看到或点击赞助商品时，需要将该交互与放置该商品的原始拍卖进行匹配。拍卖记录携带广告服务器的投放决策上下文；交互事件携带结果。将两者结合，才能将原始点击转化为可计费的费用，并关闭向我们广告投放系统的反馈循环。

This join has to happen in near-real time and within a bounded window. We limit it to 15 minutes: interactions that arrive after that are dropped. Missing an event means an advertiser action goes unbilled, so the match rate is a business-critical metric.

此关联操作必须在近实时且在一个有界窗口内完成。我们将其限制为15分钟：在此之后到达的交互将被丢弃。遗漏事件意味着广告主操作未被计费，因此匹配率是一个业务关键指标。

This article focuses on the near-real-time stream that supports ad server logic. Correct billing and campaign reporting also rely on a separate batch data pipeline. Together they form a near-classical lambda architecture, but that is a topic for another post.

本文重点介绍支持广告服务器逻辑的近实时流。正确的计费和活动报告也依赖于一个独立的批处理数据管道。它们共同构成了一个近乎经典的 lambda 架构，但这将是另一篇文章的主题。

The homegrown solution was developed more than 7 years ago at an ad-tech startup later acquired by Zalando. It was simple and worked, but accumulated limitations that improvements alone couldn't fix.

这个自研解决方案是在 7 年多前的一家广告科技初创公司开发的，该公司后来被 Zalando 收购。它很简单且有效，但积累了仅靠改进无法解决的局限性。

Homegrown solution

自研解决方案

It was a distributed Java application, which consumed Kinesis and Nakadi (abstraction on top of Kafka) streams, storing bid events in a basic in-memory cache. On each user interaction event, it searc...