Grab 的数据网格(第三部分):通过自动化 DPIs 落地数据可靠性
Introduction
简介
In the first two parts of this series, we described how Grab approaches data mesh through the Signals Marketplace: a way for teams to publish, discover, and reuse trusted data products across domains. Part II introduced the foundational tools behind certification: Hubble for metadata and ownership, Genchi for data quality observability, and the Data Contract Registry for explicit producer-consumer guarantees.
在本系列的第一篇和第二篇文章中,我们描述了 Grab 如何通过 Signals Marketplace 实践 data mesh:这是一种让团队跨域发布、发现和重用受信任数据产品的方法。第二部分介绍了认证背后的基础工具:用于元数据和所有权的 Hubble、用于数据质量可观测性的 Genchi,以及用于明确生产者-消费者保证的 Data Contract Registry。
Certification is the starting point for a trusted data marketplace. It gives downstream consumers confidence in an asset’s ownership, documentation, lineage, and quality controls. Certification does not eliminate runtime failure. A certified table can still arrive late. A certified metric can still be affected by a broken dependency. A certified Kafka stream can still violate a freshness expectation.
认证是受信任数据市场的起点。它让下游消费者对资产的所有权、文档、血缘和质量控制充满信心。认证并不能消除运行时故障。经过认证的表仍可能延迟到达。经过认证的指标仍可能受到依赖项损坏的影响。经过认证的 Kafka 流仍可能违反新鲜度期望。
Keeping certified data products reliable in production requires more than defining standards upfront. Teams need a consistent way to detect failures, diagnose the root cause, fix the issue, and verify recovery. That is where Data Production Issues (DPIs) come in. At Grab, DPIs turn data quality signals into an operational workflow.
保持认证数据产品在生产环境中的可靠性,不仅仅需要预先定义标准。团队需要一种一致的方法来检测故障、诊断根本原因、修复问题并验证恢复情况。这就是 数据生产问题(DPI)发挥作用的地方。在 Grab,DPI 将数据质量信号转化为运营工作流。
The DPI lifecycle
DPI 生命周期
A good DPI should be clear enough to act on, and it should close automatically when the underlying condition recovers. From the beginning, we designed the DPI lifecycle to be automated, with minimal human-in-the-loop.
一个好的DPI应该足够清晰以指导行动,并且在底层状态恢复时应自动关闭。从一开始,我们就将DPI生命周期设计为自动化,尽可能减少人工介入。
The lifecycle starts when Kinabalu, Grab’s incident orchestrator, observes that a data asset may no longer satisfy its contract. The contract captu...