使用OpenSearch驱动十亿规模的向量搜索

At Uber, our systems handle massive amounts of data daily, from ridesharing to delivery. We’ve traditionally used keyword-based search with Apache Lucene™. However, we needed to move beyond simple keyword matching to semantic search to understand the meaning behind searches. 

在Uber，我们的系统每天处理大量数据，从拼车到配送。我们传统上使用基于关键字的搜索与Apache Lucene™。然而，我们需要超越简单的关键字匹配，转向语义搜索，以理解搜索背后的含义。

To achieve this, we adopted Amazon® OpenSearch as our vector search engine. Its scalability, performance, and flexibility were key factors in our decision. This blog post explores our journey of evaluating and implementing OpenSearch for large-scale vector search, focusing specifically on the infrastructure challenges and solutions we encountered.

为此，我们采用了Amazon® OpenSearch作为我们的向量搜索引擎。其可扩展性、性能和灵活性是我们决策的关键因素。本文探讨了我们评估和实施OpenSearch进行大规模向量搜索的旅程，特别关注我们遇到的基础设施挑战和解决方案。

Our infrastructure for semantic search began with Apache Lucene and its HNSW (Hierarchical Navigable Small World) algorithm. We were initially excited about the prospect of incorporating vector embeddings from our machine learning models to power semantic retrieval. Early prototypes demonstrated the potential to significantly improve user experiences.

我们的语义搜索基础设施始于Apache Lucene及其HNSW（层次可导航小世界）算法。我们最初对将机器学习模型的向量嵌入纳入语义检索的前景感到兴奋。早期原型展示了显著改善用户体验的潜力。

However, as our use cases expanded and our data grew, we met several roadblocks with Lucene’s HNSW approach. We found ourselves limited by the lack of algorithm options, which hindered our ability to fine-tune tradeoffs for different scenarios. This meant we couldn’t always provide our users with the most accurate results or cost-efficient options. Furthermore, the absence of native GPU support became a performance bottleneck when dealing with the high-dimensional vectors required for complex tasks like personalized recommendations and fraud detection. This led to slower response times and limited the potential capabilities of the machine learning models we could deploy.

然而，随着我们的用例扩展和数据增长，我们在Lucene的HNSW方法上遇到了几个障碍。我们发现自己受到算法选项缺乏的限制，这妨碍了我们在不同场景中微调权衡的...