从DynamoDB迁移Uber的Ledger数据的一万亿条目到LedgerStore

Last week, we explored LedgerStore (LSG) – Uber’s append-only, ledger-style database. This week, we’ll dive into how we migrated Uber’s business-critical ledger data to LSG. We’ll detail how we moved more than a trillion entries (making up a few petabytes of data) transparently and without causing disruption, and we’ll discuss what we learned during the migration.

上周，我们探索了LedgerStore（LSG）- Uber的追加式账本数据库。本周，我们将深入探讨如何将Uber的业务关键账本数据迁移到LSG。我们将详细介绍如何透明地迁移超过一万亿条记录（占据几个PB的数据），而不会造成中断，并讨论我们在迁移过程中学到的东西。

History

历史

Gulfstream is Uber’s payment platform. It was launched in 2017 using DynamoDB for storage. At Uber’s scale, DynamoDB became expensive. Hence, we started keeping only 12 weeks of data (i.e., hot data) in DynamoDB and started using Uber’s blobstore, TerraBlob, for older data (i.e., cold data). TerraBlob is similar to AWS S3.

Gulfstream是Uber的支付平台。它于2017年使用DynamoDB进行存储发布。在Uber的规模下，DynamoDB变得昂贵。因此，我们开始只在DynamoDB中保留12周的数据（即热数据），并开始使用Uber的blobstore TerraBlob存储较旧的数据（即冷数据）。TerraBlob类似于AWS S3。

For a long-term solution, we wanted to use LSG. It was purpose-built for storing payment-style data. Its key features are:

对于长期解决方案，我们希望使用LSG。它是专为存储类似支付数据而构建的。其主要特点包括：

It is verifiably immutable (i.e., you can check that records have not been altered using cryptographic signatures)
它是可验证的不可变的（即，您可以使用加密签名检查记录是否已被更改）
Tiered storage to manage cost (the hot data is kept at a place that is best to serve requests and cold data is stored at a place that is optimized for storage)
分层存储以管理成本（热数据存储在最适合提供请求的位置，冷数据存储在最优化存储的位置）
Better lag for eventually consistent secondary indexes
更好的延迟处理一致性次要索引

So, by 2021, Gulfstream was using a combination of DynamoDB, TerraBlob, and LSG to store data.

因此，到2021年，Gulfstream使用DynamoDB、TerraBlob和LSG的组合来存储数据。

DynamoDB for the last 12 weeks of data
使用DynamoDB存储最近12周的数据
TerraBlob, Uber’s internal blob store, for cold data
TerraBlob，Uber 的内部 Blob 存储，用于冷数据
LSG, where we were writing data, and wanted to migrate to it
LSG，我们正在写入数据，并希望迁移到它

LSG is better suited for storing ledger-style data because of its immutability. The r...