eBay的通知流平台。eBay如何大规模地处理实时推送通知

Skip to main content

跳到主要内容

Company Information

公司信息

Streaming systems in event-driven microservice architecture are often compared with distributed data processing frameworks, such as Flink and Spark etc., in terms of technical design choices. At eBay, we have built a high-throughput event streaming platform in microservices architecture where eBay’s partners can subscribe to eBay's business events such as product feeds updates, bid status changes and item shipment information, and receive push notifications in real time. Here’s a high-level illustration of the data flow:

事件驱动的微服务架构下的流媒体系统在技术设计的选择上经常与分布式数据处理框架，如Flink和Spark等进行比较。在eBay，我们在微服务架构中建立了一个高吞吐量的事件流平台，eBay的合作伙伴可以订阅eBay的业务事件，如产品信息更新、投标状态变化和物品运输信息，并实时接收推送通知。以下是数据流的高层图示。

Figure 1: The high level data flow of the business event streaming

图1：业务事件流的高层次数据流

1. The external event subscriber creates the notification subscription through the eBay Notification Public API.
2. 外部事件订阅者通过eBay Notification Public API创建通知订阅。
3. The subscription is saved as part of the notification configuration metadata.
4. 订阅被保存为通知配置元数据的一部分。
5. The event producer publishes the event through RESTful API or message queue.
6. 事件生产者通过RESTful API或消息队列发布事件。
7. The raw event is delivered to the event streaming platform.
8. 原始事件被传递到事件流平台。
9. The event streaming platform resolves the subscriptions from the metadata store and performs complex event processing.
10. 事件流平台解决来自元数据存储的订阅，并执行复杂的事件处理。
11. The event is delivered to the event subscriber via HTTP push.
12. 该事件通过HTTP推送传递给事件订阅者。

Broadcast notification was a significant, though not the only, challenge during this project. The domain team requires our event streaming platform to push broadcast notifications to 20,000 subscribers in real time. For unicast notification, we can fetch the subscription from the database, process the notification and dispatch it to the recipient. However, for broadcast notification, loading all 20,000 subscriptions from the database and processing them is a time-consuming task which also blocks the event processing thread until it finishes, thus increasing the...