Hugo 的演进：使用 Apache Flink 构建 Grab 统一的一键式数据接入平台
Introduction
简介
Data drives every decision we make at Grab. As our operations scale, so does our need for robust, real-time data ingestion and processing frameworks. Enter Hugo: our self-service data platform that has long empowered teams to seamlessly route data into our Data Lake. Today, Hugo is evolving. We have taken previously siloed onboarding workflows and transformed them into one seamless, unified journey to truly democratize data ingestion and maximize efficiency.
在 Grab，数据驱动着我们做出的每一个决策。随着业务规模的扩大，我们对稳健的实时数据接入和处理框架的需求也随之增长。Hugo 应运而生：作为我们的自助数据平台，它长期以来赋能团队将数据无缝路由到我们的数据湖中。如今，Hugo 正在不断演进。我们将以前孤立的接入工作流程转变为无缝、统一的旅程，以真正实现数据接入的民主化并最大化效率。
In this blog, we’ll share how Hugo turns complex engineering hurdles into a frictionless, self-service reality. By moving away from siloed workflows, we’ve achieved a unified pipeline experience where one-click RDS CDC and self-service Kafka ingestion are the new standard.
在这篇博客中，我们将分享 Hugo 如何将复杂的工程难题转化为无缝的自助式现实。通过打破孤立的工作流，我们实现了统一的管道体验，让一键式 RDS CDC 和自助式 Kafka 接入成为新标准。
Background
背景
Figure 1. Hugo - Ingests data from every source into Grab's data lake.
图 1. Hugo - 将来自各个数据源的数据摄取到 Grab 的数据湖中。
Hugo was originally designed as a self-service platform for batch-oriented data ingestion into the Data Lake, built on a single computation engine, Spark. It provided a centralized and streamlined onboarding experience for data sources such as MySQL, Aurora, PostgreSQL, and DynamoDB.
Hugo最初被设计为一个自助服务平台，用于将面向批处理的数据摄取到数据湖中，构建在单一计算引擎Spark之上。它为MySQL、Aurora、PostgreSQL和DynamoDB等数据源提供了集中且简化的接入体验。
As the organization’s data platform evolved toward near real-time ingestion, Hugo expanded to support streaming pipelines from Kafka and MySQL binlog. This evolution introduced a more distributed architecture, where ingestion workflows spanned multiple systems, including Kafka Connect, Sprinkler (an in-house Go-based S3 writer), and Hugo.
随着组织的数据平台向近乎实时摄取演进，Hugo 扩展了对来自 Kafka 和 MySQL binlog 的流式管道的支持。这一演进引入了更加分布式的架构，其中摄取工作流跨越了多个系统，包括 Kafka Connect、Sprinkler（一个内部基于 Go 的 S3 写入器）和 Hugo。