在Zookeeper集群之间透明地迁移Kafka

Learn more about how to migrate your Kafka cluster from one Zookeeper cluster to another without any user impact.

了解更多关于如何将你的Kafka集群从一个Zookeeper集群迁移到另一个集群，而不影响用户。

Introduction

简介

Kafka is an open-source distributed event-streaming platform. It depends on Zookeeper, another open-source distributed coordination system, to store cluster metadata. At Airbnb, Kafka forms the backbone of our data infrastructure, powering use cases such as event logging and change data capture that help us better understand our guests and hosts, and make decisions that improve our product.

Kafka是一个开源的分布式事件流平台。它依赖于另一个开源的分布式协调系统Zookeeper来存储集群元数据。在Airbnb，Kafka构成了我们数据基础设施的骨干，为事件记录和变化数据采集等用例提供动力，帮助我们更好地了解我们的客人和房东，并做出改进我们产品的决定。

We run several production Kafka clusters, the largest of which being our oldest cluster consisting of hundreds of brokers and supporting over 1GB/s of incoming traffic. Prior to our migration, this Kafka cluster relied on a legacy Zookeeper setup — a multi-tenant Zookeeper cluster shared between many different production use cases, which means that any incident on the cluster would affect all dependent services, including Kafka. Moreover, this Zookeeper cluster also lacks a clear owner. We wanted to migrate Kafka out of this Zookeeper cluster into a separate, dedicated Zookeeper cluster with clear ownership and better isolation from other use cases.

我们运行着几个生产型Kafka集群，其中最大的一个是我们最古老的集群，由数百个经纪人组成，支持超过1GB/s的传入流量。在我们迁移之前，这个Kafka集群依赖于一个传统的Zookeeper设置--一个由许多不同的生产用例共享的多租户Zookeeper集群，这意味着集群上的任何事件都会影响所有的依赖服务，包括Kafka。此外，这个Zookeeper集群也缺乏一个明确的所有者。我们想把Kafka从这个Zookeeper集群中迁移到一个独立的、专门的Zookeeper集群中，它有明确的所有权，并与其他用例有更好的隔离。

The goal was for the migration to be done transparently, without any data loss, downtime or impact to Kafka users, as well as other Zookeeper users.

我们的目标是让迁移透明地进行，没有任何数据损失、停机或对Kafka用户以及其他Zookeeper用户的影响。

The recently released Kafka 2.8.0 includes changes for KIP-500, the community’s proposal to remove Kafka’s Zookeeper dependency. Since this feature in Kafka 2.8.0 is neither complete nor production ...