Apache Flink® 在 Kubernetes 上

Airbnb’s Use of A New Flink platform evolved from Apache Hadoop® Yarn

Airbnb使用的新Flink平台是从Apache Hadoop® Yarn演变而来的

Introduction

介绍

At Airbnb, Apache Flink was introduced in 2018 as a supplementary solution for stream processing. It ran alongside Apache Spark™ Streaming for several years before transitioning to become the primary stream processing platform. In this blog post, we will delve into the evolution of Flink architecture at Airbnb and compare our prior Hadoop Yarn platform with the current Kubernetes-based architecture. Additionally, we will discuss the efforts undertaken throughout the migration process and explore the challenges that arose during this journey. In the end we will summarize the impact, learnings along the way and future plans.

在Airbnb，Apache Flink于2018年作为流处理的补充解决方案引入。它与Apache Spark™ Streaming并行运行了几年，然后过渡为主要的流处理平台。在本博客文章中，我们将深入探讨Airbnb的Flink架构的演变，并将我们之前的Hadoop Yarn平台与当前的基于Kubernetes的架构进行比较。此外，我们还将讨论在迁移过程中所采取的努力，并探讨在这一过程中出现的挑战。最后，我们将总结影响、经验以及未来计划。

Architecture Evolution

架构演进

The evolution of Airbnb’s streaming processing architecture based on Apache Flink can be categorized into three distinct phases:

基于Apache Flink的Airbnb流处理架构的演变可以分为三个不同的阶段：

Phase One: Flink jobs operated on Hadoop Yarn with Apache Airflow serving as the job scheduler.

第一阶段：Flink 作业在 Hadoop Yarn 上运行，Apache Airflow 作为作业调度程序。

Around 2018, several teams at Airbnb adopted Flink as their streaming processing engine, mainly due to its superior low-latency capabilities compared to Spark Streaming. During this period, Flink jobs were running on Hadoop Yarn, and Airflow was employed as the workflow manager for task scheduling and dependency management.

大约在2018年，Airbnb的几个团队采用了Flink作为其流处理引擎，主要是因为其相对于Spark Streaming具有卓越的低延迟能力。在此期间，Flink作业在Hadoop Yarn上运行，并且使用Airflow作为任务调度和依赖管理的工作流管理器。

The selection of Airflow as the workflow manager was largely influenced by its widespread use in addressing various job scheduling needs, as there were no other user-friendly open-source alternatives readily available at that time. Each team was responsible f...