介绍 uFowarder：Kafka 异步队列的消费者代理

Uber has one of the largest deployments of Apache Kafka® in the world, processing trillions of messages and multiple petabytes of data per day. Three years ago, we built a push-based consumer proxy for Kafka’s Async Queue. It’s become the primary option for reading data from Kafka in pub-sub use cases at Uber, with over 1,000 consumer services onboarded. We open-sourced the solution under the name uForwarder.

Uber 拥有世界上最大的 Apache Kafka® 部署之一，每天处理万亿条消息和多个 PB 的数据。三年前，我们为 Kafka 的异步队列 构建了一个基于推送的消费者代理。它已成为 Uber 在发布-订阅用例中从 Kafka 读取数据的主要选项，已有超过 1,000 个消费者服务接入。我们以 uForwarder 的名称开源了该解决方案。

This blog describes the challenges we faced when productionizing uForwarder and the solutions we implemented to optimize hardware efficiency, ensure consumer isolation, address head-of-line blocking, and support message delay processing. By reading this blog, you can understand the thinking behind uForwarder before applying it to your use cases.  

本文描述了我们在将 uForwarder 投入生产时面临的挑战，以及我们实施的解决方案，以优化硬件效率、确保消费者隔离、解决 head-of-line blocking，并支持消息延迟处理。通过阅读本文，您可以在将 uForwarder 应用于您的用例之前理解其背后的思路。  

We broke down the motivation and high-level solutions for uForwarder in a previous blog. In short, we encountered problems like partition scalability, head-of-line blocking, and support issues for multiple programming languages.  

我们在之前的 博客 中分解了 uForwarder 的动机和高层次解决方案。简而言之，我们遇到了分区可扩展性、头阻塞以及多编程语言支持等问题。  

We developed Consumer Proxy, a novel push proxy, to solve those challenges. At a high level, the Consumer Proxy cluster:

我们开发了 Consumer Proxy，这是一种新型的 push proxy，用来解决这些挑战。从高层来看，Consumer Proxy 集群：

1. Fetches messages from Kafka using Kafka’s binary protocol.
2. 使用 Kafka 的二进制协议从 Kafka 获取消息。
3. Pushes each message separately to a consumer service instance that exposes a gRPC® service endpoint. The consumer service processes each message separately, and sends the result back to the Consumer Proxy cluster.
4. 将每条消息单独推送到暴露 gRPC® 服务端点的消费者服务实例。消费者服务单独处理每条消息，并将结果发送回 Consumer Proxy 集群。
5. Receives the gRPC status code from the consumer service.  
6. 从消费者服务接收 gRPC 状态码。  
7. Aggregates the cons...