如何在 OpenSearch™ 中使用基于拉取的摄取来索引 Uber 的流数据

At Uber, our business operates in real time. Whether you’re hailing a ride, ordering from a restaurant, or tracking a delivery, search is the critical starting point. Our search platform powers these experiences at a massive scale, indexing everything from restaurant menus and destinations to the live locations of drivers and couriers. 

在 Uber，我们的业务实时运作。无论您是在叫车、从餐厅点餐还是跟踪配送，搜索都是关键的起点。我们的搜索平台在大规模上支持这些体验，索引从餐厅菜单和目的地到司机和快递员的实时位置的所有内容。

Given its central role, our search platform must meet stringent demands for performance, scalability, and data freshness. To achieve this, its architecture was built on two foundational principles: a pull-based ingestion model and an active-active deployment. The pull-based model, built on Apache Kafka**®,** decouples data producers from the search cluster, allowing our platform to ingest data at its own pace for greater reliability. This is paired with an active-active architecture where we operate in multiple regions, keeping each index fresh to support seamless failovers. By consuming from Uber’s cross-replicated Kafka infrastructure, the platform maintains a consistent global view, ensuring the high availability critical to our operations.

鉴于其核心角色，我们的搜索平台必须满足对性能、可扩展性和数据新鲜度的严格要求。为此，其架构建立在两个基础原则之上：基于拉取的摄取模型和主动-主动部署。基于拉取的模型，建立在 Apache Kafka**®**之上，将数据生产者与搜索集群解耦，使我们的平台能够以自己的节奏摄取数据，从而提高可靠性。这与主动-主动架构相结合，我们在多个区域运营，保持每个索引的新鲜度，以支持无缝故障转移。通过从 Uber 的跨复制 Kafka 基础设施中消费，该平台保持一致的全球视图，确保我们运营所需的高可用性。

In this blog, we’ll take you on a deep dive into the pull-based ingestion model that powers our architecture. We’ll share the journey of contributing pull-based indexing to OpenSearch and explain how this enables us to migrate our in-house search systems to the open-source platform. More details on the evolution of Uber’s search platform can be found in a prior blog.

在这篇博客中，我们将深入探讨驱动我们架构的基于拉取的摄取模型。我们将分享将基于拉取的索引贡献给OpenSearch的历程，并解释这如何使我们能够将内部搜索系统迁移到开源平台。有关Uber搜索平台演变的更多细节可以在之前的博客中找到。

To understand why we adopted a pull-based model, it’s helpful to first examine the limitations of the push-based architectur...