扩展 Grab 的数据湖：我们采用 Apache Iceberg 的历程
Introduction: The evolution of Grab’s Data Lake
简介：Grab 数据湖的演进
At Grab’s scale, managing petabytes of data across billions of S3 objects demands more than a storage layer. It demands a robust architectural primitive that supports the high-concurrency needs of a modern “Lakehouse.” Our goal is full storage-compute separation, leveraging S3 as an elastic foundation for both near-real-time metrics and large-scale batch transformations.
在 Grab 的规模下，管理数十亿个 S3 对象中的 PB 级数据需要的不仅仅是一个存储层。它需要一个强大的架构原语，以支持现代“Lakehouse”的高并发需求。我们的目标是实现完全的存储计算分离，利用 S3 作为弹性基础，同时支持近实时指标和大规模批处理转换。
For years, the vast majority of our tables were Hive Parquet, managed through the Hive Metastore with a directory-based layout. This model served us well, but as data volume grew, the directory-and-metastore approach became the limiting factor. We are now transitioning to a table-centric architecture built on modern table formats, treating data as a first-class primitive to ensure consistency and performance across our internal data transformation platforms: Slide, which powers batch transformations, and Hugo, which handles online-to-data-lake ingestion. Along the way, we also built the UnifiedSparkCatalog, a unified Spark catalog that hides table-format differences from users entirely, which we are open-sourcing alongside this post.
多年来，我们绝大多数的表都是 Hive Parquet，通过 Hive Metastore 以基于目录的布局进行管理。这种模式一直很好地服务于我们，但随着数据量的增长，目录和 Metastore 的方法成为了限制因素。我们现在正过渡到建立在现代表格式之上的以表为中心的架构，将数据视为一等原语，以确保我们内部数据转换平台的一致性和性能：Slide 负责批处理转换，Hugo 处理在线到数据湖的摄取。在此过程中，我们还构建了 UnifiedSparkCatalog，这是一个统一的 Spark catalog，完全向用户隐藏了表格式的差异，我们将在这篇文章中同时将其开源。
The catalyst for change: Challenges with Hive Parquet
变革的催化剂：Hive Parquet 面临的挑战
For years, Hive Parquet was the backbone of our Data Lake, representing the vast majority of our tables. However, as data volume scaled, the architectural limitations of directory-based storage became apparent. We identified four primary bottlenecks:
多年来，Hive Parquet 一直是我们数据湖的骨干，占我们表的绝大多数。然而，随着数据量的扩展，基于目录的存储的架构局限性变得明显。我们确定了四个主要瓶颈：
- Catalog latency: The Hive Metastore ...