在HBase上构建可扩展的近实时索引

Ankita Wagh | Software Engineer, Storage and Caching

Ankita Wagh | 软件工程师，存储和缓存

HBase is one of the most critical storage backends at Pinterest, powering many of our online traffic storage services like Zen (graph database) and UMS (wide column data store). Although HBase has many advantages like strong consistency at row level in high volume requests, flexible schema, low latency access to data, and Hadoop integration, it doesn’t natively support advanced indexing and querying. Secondary indexing is one of the most demanded features by our clients, but supporting that directly in HBase is quite challenging. Maintaining separate index tables as the number of indexes grows is not a scalable solution in terms of query efficiency and code complexity. This motivated us to build a storage solution called Ixia, which provides near real-time secondary indexing on HBase. The design is largely inspired by Lily HBase Indexer.

HBase是Pinterest最关键的存储后端之一，为我们的许多在线流量存储服务提供动力，如Zen（图形数据库）和UMS（宽列数据存储）。虽然HBase有很多优点，比如在大批量请求中行级的强一致性，灵活的模式，对数据的低延迟访问，以及Hadoop集成，但它并不支持高级索引和查询。二级索引是我们的客户最需要的功能之一，但在HBase中直接支持二级索引是相当具有挑战性的。随着索引数量的增加，维护单独的索引表在查询效率和代码复杂性方面不是一个可扩展的解决方案。这促使我们建立了一个名为Ixia的存储解决方案，它在HBase上提供近乎实时的二级索引。该设计主要是受Lily HBase Indexer的启发。

Ixia provides a generic search interface on top of HBase, which plays the role of the source-of-truth database. A search engine is natively optimized for inverted index lookup, stores indexes. Search Engines also provide a rich set of search and aggregation queries and support the majority of indexing and retrieval use cases at Pinterest.

Ixia在HBase之上提供了一个通用的搜索接口，HBase扮演着真理之源数据库的角色。搜索引擎对倒置的索引查询进行了原生优化，存储了索引。搜索引擎还提供丰富的搜索和聚合查询，并支持Pinterest的大多数索引和检索用例。

In this post, we will first provide a brief introduction on the architecture of the system and the design choices. We will then cover how we maintain Ixia’s data consistency SLA, how we address disaster recovery, and what measures we had to take to improve the overall system performance. Finally, we will provide some guidelines regarding future work and opport...