LLM辅助的向量相似性搜索

As the complexity of data retrieval requirements continue to grow, traditional search methods often struggle to provide relevant and accurate results, especially for nuanced or conceptual queries. Vector similarity search has emerged as a powerful technique for finding semantically similar information. It refers to finding vectors in a large dataset that are most similar to a given query vector, typically using some distance or similarity measure. The concept originated in the 1960s with the work by Minsky and Papert on nearest neighbour search 1. Since then, the idea has evolved substantially with modern approaches often using approximate methods to enable fast search in high-dimensional spaces, such as locality-sensitive hashing 2 and graph-based indexing 3.

随着数据检索需求的复杂性不断增加，传统的搜索方法往往难以提供相关和准确的结果，尤其是对于细微或概念性的查询。向量相似性搜索已成为查找语义相似信息的强大技术。它指的是在大型数据集中找到与给定查询向量最相似的向量，通常使用某种距离或相似性度量。这个概念起源于1960年代Minsky和Papert关于最近邻搜索的工作1。从那时起，这一想法已经有了实质性的演变，现代方法通常使用近似方法来实现高维空间中的快速搜索，例如局部敏感哈希2和基于图的索引3。

Recently, vector similarity search has become a crucial component in many machine learning and information retrieval applications. It is one of the key technologies that popularised the idea of Retrieval Augmented Generation (RAG) 4 which increased the applicability of Transformer 5 based Generative Large Language Models (LLMs) 6 in domain-specific tasks without requiring any further training or fine-tuning. However, the effectiveness of the vector search can be limited when dealing with intricate queries or contextual nuances. For example, from a typical vector similarity search perspective, “I like fishing” and “I do not like fishing” may be quite close to each other, while in reality, they are the exact opposite. In this blog post, we discuss an approach that we experimented with that combines vector similarity search with LLMs to enhance the relevance and accuracy of search results for such complex and nuanced queries. We leverage the strengths of both techniques: vector similarity search for efficient shortlisting of potential matc...