GraphLoad:一个以性能和一致性加载和更新超过100亿顶点图形的框架

NuGraph is a graph database platform developed at eBay that is cloud-native, scalable and performant. It is built upon the open-source graph database JanusGraph, with FoundationDB as the backend that stores graph elements and indexes. This article presents GraphLoad, a utility developed at eBay to load and update graphs at scale. GraphLoad is deployed in production to load a graph that has grown to over 10 billion vertices, loading both real-time streaming data and as batch data from files, while continuously serving real-time graph queries. 

NuGraph是eBay开发的一个图形数据库平台，具有云原生性、可扩展性和高性能。它建立在开源图数据库JanusGraph的基础上，以FoundationDB作为存储图元素和索引的后端。本文介绍了GraphLoad，这是eBay开发的一个工具，用于大规模加载和更新图形。GraphLoad被部署在生产中，用于加载一个已经增长到超过100亿个顶点的图，同时加载实时流数据和文件中的批处理数据，并持续提供实时图查询。

GraphLoad provides a generic solution to load different source data layouts and graph schemas. Based on the distributed transaction support from NuGraph, it provides exactly-once loading and updating guarantees under various failure conditions. GraphLoad performs parallel loading with hundreds of loaders located in multiple data centers, with scalability and high availability supported by NuGraph.

GraphLoad提供了一个通用的解决方案来加载不同的源数据布局和图形模式。基于NuGraph的分布式事务支持，它在各种故障条件下提供了精确的一次加载和更新保证。GraphLoad通过位于多个数据中心的数百个加载器执行并行加载，并由NuGraph支持可扩展性和高可用性。

Motivation and Challenges 

动机和挑战

Many applications at eBay, including eBay customer-facing applications, internal business operations and IT management operations, require Graph Databases (GraphDBs) to capture and make use of business insights derived from diverse data sources. In a GraphDB, business insights are stored as a graph, with vertices representing business entities and edges representing the relationships between the business entities. Business insights are continuously loaded and updated in order to assist real-time decision making. 

eBay的许多应用，包括eBay面向客户的应用、内部业务运营和IT管理运营，都需要图形数据库（GraphDB）来捕捉和利用来自不同数据源的业务洞察力。在GraphDB中，业务洞察力以图的形式存储，其中顶点代表业务实体，边代表业务实体之间的关系。业务洞察力被不断加载和更新，以协助实时决策。

A number of difficulties make developing a scalable grap...