介绍uGroup。Uber的消费者管理框架

Background

背景介绍

Apache Kafka® is widely used across Uber’s multiple business lines. Take the example of an Uber ride: When a user opens up the Uber app, demand and supply data are aggregated in Kafka queues to serve fare calculations. When a ride request is accepted by a driver, push notifications in Kafka queue are sent to mobile devices. After a ride is finished, post-trip processing, including payment and receipt sending, leverages Kafka. During the entire operation, the data and messages flowing between services are also ingested into Apache Hive™ for data analytic purposes. In a word, Apache Kafka is a critical service that empowers Uber’s business.

Apache Kafka®被广泛应用于Uber的多条业务线。以Uber乘车为例。当用户打开Uber应用时，需求和供应数据被聚集在Kafka队列中，为票价计算服务。当司机接受乘车请求时，Kafka队列中的推送通知被发送到移动设备。乘坐结束后，出行后的处理，包括支付和收据发送，都是利用Kafka。在整个操作过程中，服务之间流动的数据和消息也被摄入Apache Hive™中，用于数据分析目的。总而言之，Apache Kafka是一个重要的服务，它赋予了Uber的业务。

Given its high popularity, we are operating large scale Kafka clusters across multiple regions. We started our Kafka journey in early 2015 with a few-node Kafka cluster in one region. With the tremendous growth of Uber’s business and expansion of Kafka usages, we ran into scaling and operational issues, and got many interesting user requests from customers.

鉴于其高度普及，我们在多个地区运营大规模的Kafka集群。我们在2015年初开始了我们的Kafka之旅，当时在一个地区只有几个节点的Kafka集群。随着Uber业务的巨大增长和Kafka使用范围的扩大，我们遇到了扩展和运营问题，并从客户那里得到了许多有趣的用户请求。

One of the most common issues we have run into is how to efficiently monitor the state of a large number of consumers. Having evaluated many open source solutions, with the large scale and unique setup, we finally decided to build a new observability framework for monitoring the state of Kafka consumers. Today, we are delighted to introduce uGroup, our internal Kafka consumer monitoring service.

我们遇到的最常见的问题之一是如何有效地监控大量消费者的状态。在评估了许多开源的解决方案后，由于规模大、设置独特，我们最终决定建立一个新的可观察性框架来监控Kafka消费者的状态。今天，我们很高兴地介绍uGroup，我们内部的Kafka消费者监控服务。

Challenges

挑战

At Uber’s scale, improving Kafka consumer observability can greatly simplify the process of problem detection, debuggin...