数据网 - 一个数据移动和处理平台 @ Netflix

By Bo Lei, Guilherme Pires, James Shao, Kasturi Chatterjee, Sujay Jain, Vlad Sydorenko

作者：雷波，Guilherme Pires，James Shao，Kasturi Chatterjee，Sujay Jain，Vlad Sydorenko

Background

背景介绍

Realtime processing technologies (A.K.A stream processing) is one of the key factors that enable Netflix to maintain its leading position in the competition of entertaining our users. Our previous generation of streaming pipeline solution Keystone has a proven track record of serving multiple of our key business needs. However, as we expand our offerings and try out new ideas, there’s a growing need to unlock other emerging use cases that were not yet covered by Keystone. After evaluating the options, the team has decided to create Data Mesh as our next generation data pipeline solution.

实时处理技术（A.K.A流处理）是使Netflix在娱乐用户的竞争中保持领先地位的关键因素之一。我们的上一代流媒体管道解决方案Keystone在服务我们的多个关键业务需求方面有着良好的记录。然而，随着我们扩展产品和尝试新的想法，我们越来越需要解开Keystone尚未覆盖的其他新兴用例。在评估了各种选择后，团队决定创建Data Mesh作为我们的下一代数据管道解决方案。

Last year we wrote a blog post about how Data Mesh helped our Studio team enable data movement use cases. A year has passed, Data Mesh has reached its first major milestone and its scope keeps increasing. As a growing number of use cases on board to it, we have a lot more to share. We will deliver a series of articles that cover different aspects of Data Mesh and what we have learned from our journey. This article gives an overview of the system. The following ones will dive deeper into different aspects of it.

去年我们写了一篇博文，介绍了Data Mesh是如何帮助我们的Studio团队实现数据移动用例的。一年过去了，Data Mesh已经达到了它的第一个重要里程碑，它的范围也在不断扩大。随着越来越多的用例加入它，我们还有很多东西要分享。我们将提供一系列的文章，涵盖Data Mesh的不同方面，以及我们在旅程中的收获。这篇文章给出了该系统的概述。接下来的文章将深入探讨它的不同方面。

Data Mesh Overview

数据网概述

Previously, we defined Data Mesh as a fully managed, streaming data pipeline product used for enabling Change Data Capture (CDC) use cases. As the system evolves to solve more and more use cases, we have expanded its scope to handle not only the CDC use cases but also more general data movement and processing use cases such that:

以前，我们把Data Mesh定义为一个完全管理的...