用消费者代理启用无缝Kafka异步队列

Uber has one of the largest deployments of Apache Kafka in the world, processing trillions of messages and multiple petabytes of data per day. As Figure 1 shows, today we position Apache Kafka as a cornerstone of our technology stack. It empowers a large number of different workflows, including pub-sub message buses for passing event data from the rider and driver apps, streaming analytics (e.g., Apache Flink, Apache Samza), streaming database changelogs to the downstream subscribers, and ingesting all sorts of data into Uber’s Apache Hadoop data lake.

Uber拥有世界上最大的Apache Kafka部署之一，每天处理数万亿条消息和多个PB的数据。如图1所示，今天我们将Apache Kafka定位为我们技术栈的基石。它支持大量不同的工作流程，包括用于传递来自乘客和司机应用程序的事件数据的pub-sub消息总线、流式分析（如Apache Flink、Apache Samza）、流式数据库变更记录到下游用户，以及将各种数据摄取到Uber的Apache Hadoop数据湖。

Figure 1: Kafka Ecosystem at Uber

图1：Uber的Kafka生态系统

More than 300 microservices at Uber are leveraging Kafka for pub-sub message queueing between microservices. In the following article we will go in depth on this particular use case, and discuss both how we arrived at the Consumer Proxy, and what it has enabled.

Uber的300多个微服务都在利用Kafka在微服务之间进行pub-sub消息排队。在下面的文章中，我们将深入探讨这个特殊的用例，并讨论我们是如何得到消费者代理的，以及它所带来的好处。

Streaming vs. Messaging

流媒体与信息传递

Apache Kafka is a stream-oriented system: message ordering is assumed in the design of the system and is a fundamental semantic property provided by the system to consumers (i.e., consumers see messages in the same order that they are received by the streaming system). To ensure that any happens-before relationship encoded in the ordering is respected:

Apache Kafka是一个面向流的系统：在系统的设计中假定了消息的排序，并且是系统提供给消费者的一个基本语义属性（即消费者看到的消息与流系统收到的顺序相同）。为了确保在排序中编码的任何发生在之前的关系得到尊重。

1. Messages of the same topic partition are consumed exclusively by only one consumer of the same group (shown in Figure 2).
2. 同一主题分区的信息只被同一组的一个消费者所消费（如图2所示）。
3. The Kafka consumer recommends the following pattern:
4. Kafka消费者推荐以下模式。

a. Reads a single message from a partition

a.从一个分区中读取一条信息

b. Processes that message

b.处理该信息

c. Does not process the next message from that partition un...