在 Meta 规模下迁移数据摄入系统
- Meta’s data ingestion system, which our engineering teams leverage for up-to-date snapshots of the social graph, has recently undergone a significant revamp to enhance its reliability at scale.
- Meta 的数据摄取系统，我们的工程团队利用它来获取 social graph 的最新快照，最近经历了重大改造，以提升其在大规模下的可靠性。
- Moving from our legacy system to our new architecture required a large-scale migration of our entire data ingestion system.
- 从我们的遗留系统迁移到新架构，需要对整个数据摄取系统进行大规模迁移。
- We’re sharing the solutions and strategies that enabled a successful large-scale system migration, as well as the key factors that influenced our architectural decisions.
- 我们正在分享实现了成功大规模系统迁移的解决方案和策略，以及影响我们架构决策的关键因素。
At Meta, our social graph is powered by one of the largest MySQL deployments in the world. Every day, our data ingestion system incrementally scrapes several petabytes of social graph data from MySQL into the data warehouse to power the analytics, reporting, and downstream data products that teams across the company utilize for tasks ranging from day-to-day decision-making to machine learning model training and product development.
在 Meta，我们的 social graph 由世界上最大的 MySQL 部署之一提供支持。每天，我们的数据摄取系统都会从 MySQL 增量抓取数 PB 的 social graph 数据到 data warehouse，以支持公司各团队用于从日常决策到机器学习模型训练和产品开发的各种任务的 analytics、reporting 和下游数据产品。
We’ve recently revamped our data ingestion system’s architecture to significantly enhance its efficiency and reliability. The new architecture moves away from customer-owned pipelines, which functioned effectively at a small scale, to a simpler self-managed data warehouse service that still operates efficiently at hyperscale.
我们最近彻底改造了数据摄取系统的架构，以显著提升其效率和可靠性。新架构摒弃了客户拥有的管道，这些管道在小规模下运行良好，转向一个更简单的自管理数据仓库服务，该服务在 hyperscale 下仍能高效运行。
We’ve successfully transitioned 100% of the workload and fully deprecated the legacy system. But migrating a data ingestion system of this scale was a major challenge. Several important solutions and strategies helped make a migration of this scope successful.
我们已成功迁移 100% 的工作负载，并完全弃用遗留系统。但是，迁移这种规模的数据摄取系统是一个重大挑战。几个重要的解决方案和策略帮助使这种规模的迁移成功。