重建 Uber 的 Apache Pinot™ 查询架构

Built on Apache Pinot™, the real-time analytics platform at Uber powers hundreds of critical use cases, from user-facing analytics and experiences to massive internal platforms like log search, tracing, and more. Until 2024, nearly all of the hundreds of millions of daily Pinot queries were served from Neutrino, an internal fork of Presto® optimized for low latency and high QPS. While our Neutrino-based query architecture has served us well over the years, we’ve also faced significant challenges in building a coherent query system over this layered architecture.

基于 Apache Pinot™，Uber 的实时分析平台支撑着数百个关键用例，从面向用户的分析和体验到日志搜索、链路追踪等大规模内部平台。直到 2024 年，每天数亿条 Pinot 查询几乎全部由 Neutrino 提供，Neutrino 是 Presto® 的一个内部分支，针对低延迟和高 QPS 进行了优化。尽管基于 Neutrino 的查询架构多年来一直表现良好，但我们也面临着在这种分层架构之上构建统一查询系统的重大挑战。

Transitioning away from Neutrino seemed extraordinarily challenging a few years ago, since Neutrino could run complex OLAP queries with multiple sub-queries and window functions at high QPS and low latency with minimal reliability risks. While Pinot’s MSE (Multi-Stage Engine) was promising, it was originally aimed at adding full Distributed SQL support, with a focus on completeness. A new MSE query could easily involve large scans or huge joins on arbitrarily large tables, which could impact a tenant’s reliability.

几年前，从 Neutrino 迁移看起来极具挑战，因为 Neutrino 能够以高 QPS 和低延迟运行带有多个子查询和窗口函数的复杂 OLAP 查询，且可靠性风险极低。虽然 Pinot 的 MSE（Multi-Stage Engine）前景可期，但它最初的目标是增加完整的 Distributed SQL 支持，重点在于完备性。一条新的 MSE 查询很容易涉及对任意大表的大规模扫描或巨大 join，从而影响租户的可靠性。

In this blog, we share how we’ve rebuilt our query architecture to begin our transition away from Neutrino. At the core of this transition is the new Pinot MSE Lite Mode, which combines our learnings from Neutrino, Pinot’s single-stage scatter-gather query engine, and the Pinot MSE. Not only is the new architecture simpler, relying solely on Apache Pinot’s query engines, it also significantly increases the overall capability of the system.

在这篇博客中，我们分享如何重建查询架构，开始逐步脱离 Neutrino。此次过渡的核心是全新的 Pinot MSE Lite Mode，它融合了我们在 Neutrino...