数据工程师 2.0. 第二部分：检索增强生成

In Part I of this series we delved into how foundational models are shaping new responsibilities for Data Engineers. We highlighted three key phases in the large language model (LLM) engineering process: retrieval augmented generation (RAG), fine-tuning and pre-training GenAI models.

在本系列的第一部分中，我们深入探讨了基础模型如何为数据工程师塑造新的职责。我们强调了大型语言模型（LLM）工程过程中的三个关键阶段：检索增强生成（RAG）、微调和预训练GenAI模型。

I currently serve as a Data Engineer at Adevinta, specialising in the Re-commerce vertical that encompasses platforms such as Leboncoin, Milanuncios, Kleinanzeigen, Marktplaats, Subito and more. My educational background is in Computer Science. I began my professional journey as a Business Intelligence consultant, transitioning to the role of a Data Scientist in 2016. Over the past six years, I have further honed my skills, primarily working as a Data Engineer.

我目前在Adevinta担任数据工程师，专注于包括Leboncoin, Milanuncios, Kleinanzeigen, Marktplaats, Subito等平台的再商业化垂直领域。我的教育背景是计算机科学。我开始我的职业生涯作为商业智能顾问，并于2016年转型为数据科学家。在过去的六年中，我进一步磨练了我的技能，主要担任数据工程师。

In Part II, our focus will be on the RAG phase. Specifically, we’ll explore the most valuable topics that I’ve discovered, sparing you hours of independent research. I’ve consolidated all the relevant information that I found here to serve as your go-to resource whenever needed. Don’t feel overwhelmed by the wealth of information; think of this blog as a centralised hub for genuinely useful insights. The aim is to provide you with pre-filtered information from the vast sea of online content, ensuring familiarisation with concepts and ready access to examples for your specific interests.

在第二部分中，我们将重点关注RAG阶段。具体来说，我们将探讨我发现的最有价值的主题，节省您数小时的独立研究。我已将所有相关信息整合在这里，以便在需要时作为您的首选资源。不要被大量信息所压倒；将此博客视为一个集中式中心，提供真正有用的见解。我们的目标是为您提供来自浩瀚在线内容的预过滤信息，确保您熟悉概念并能方便地访问与您特定兴趣相关的示例。

Image 1: The stages of GenAI & Maturity

图像 1：GenAI 的阶段与成熟度

RAG (Retrieval-Augmented Generation) systems stand out for their ability to extract relevant details from a given knowledge base, facilitating the creation of information that is not only factual but also ...