使用 Neutrino 处理数百万条 Apache Pinot™ 查询

Apache Pinot™ is a real-time OLAP database capable of ingesting data from real-time streams and offline data sources. Uber has used Pinot for over 6 years, and in that time we’ve seen a rapid increase in its adoption. Today, Pinot handles a wide spectrum of use cases at Uber, from real-time use cases with over 1 million writes per second, 100 QPS, and less than 500 ms latency, to use cases that require low-latency analytics on offline data.

Apache Pinot™ 是一个实时 OLAP 数据库，能够从实时流和离线数据源中摄取数据。Uber 使用 Pinot 已超过 6 年，在此期间我们见证了其采用的迅速增加。如今，Pinot 在 Uber 处理广泛的用例，从每秒超过 100 万次写入、100 QPS 和小于 500 毫秒延迟的实时用例，到需要离线数据低延迟分析的用例。

In this blog, we share how our platform users leverage Neutrino, an internal fork of Presto®, to query their data in Pinot.

在这篇博客中，我们分享了我们的平台用户如何利用 Neutrino（一个内部的 Presto® 分支）在 Pinot 中查询他们的数据。

Throughout this blog, queries refer to read queries,  because in Pinot we usually refer to writes as ingestion.

在整个博客中，查询指的是读取查询，因为在Pinot中我们通常将写入称为摄取。

Pinot can be queried using SQL, but it has two query engines: the V1 Engine and the more recent Multistage Engine. As of this writing, Pinot’s V1 Engine is the default engine, and the Multistage Engine has to be explicitly configured for use.

Pinot可以使用SQL进行查询，但它有两个查询引擎：V1引擎和最近的多阶段引擎。截至本文撰写时，Pinot的V1引擎是默认引擎，多阶段引擎必须显式配置才能使用。

Pinot’s V1 Engine can handle higher QPS at a much lower latency when compared with the Pinot Multistage Engine. The tradeoff is that ‌SQL features are quite limited with the V1 engine. Presto is a stateless query engine capable of reading from many different data sources via its connector architecture.

与Pinot多阶段引擎相比，Pinot的V1引擎可以在更低的延迟下处理更高的QPS。权衡是V1引擎的SQL功能非常有限。Presto是一个无状态的查询引擎，能够通过其连接器架构从许多不同的数据源读取数据。

Pinot’s V1 Engine is optimized for high QPS and low latency. The engine has a scatter-gather execution and it only supports queries with a single SELECT, WHERE, GROUP BY, and ORDER BY clause. In other words, it doesn’t support sub-queries, joins, or window functions. The engine has a custom planner that handles specific query shapes with custom operators. For...