评估驱动的开发：大规模评估GenAI的经验教训

[

The Airbnb Tech Blog

Airbnb技术博客

](https://medium.com/airbnb-engineering?source=post_page---publication_nav-53c7c27702d5-e817e5ae5788---------------------------------------)

[

The Airbnb Tech Blog

](https://medium.com/airbnb-engineering?source=post_page---post_publication_sidebar-53c7c27702d5-e817e5ae5788---------------------------------------)

Creative engineers and data scientists building a world where you can belong anywhere. http://airbnb.io

富有创造力的工程师和数据科学家正在构建一个让你在任何地方都能找到归属感的世界。http://airbnb.io

How Airbnb teams build trustworthy Generative AI products by treating evaluation as a first-class engineering discipline; not an afterthought.

Airbnb团队如何将评估视为一流的工程学科，而非事后补救，从而构建值得信赖的生成式AI产品。

A contemporary, multi-level home nestled on a steep, vegetated hillside. The exterior is completely clad in light brown vertical wood siding. A curved concrete driveway leads up to a garage on the right side of the house. The home features dark-framed windows, a lower-level balcony with red railings, and is surrounded by vibrant green trees and purple and red flowering bushes on a sunny day.

Nestled into the lush hillside, this stunning modern retreat features striking natural wood architecture, terraced balconies, and a serene landscape.

这座迷人的现代度假屋隐于郁郁葱葱的山坡之中，拥有引人注目的原木建筑、错落有致的露台以及宁静的自然风光。

By: Rohit Girme, Dan Miller, Mia Zhao, Lifan Yang, Clint Kelly

作者： Rohit Girme, Dan Miller, Mia Zhao, Lifan Yang, Clint Kelly

Introduction

简介

Generative AI breaks a lot of the assumptions that used to hold true for software testing. Unlike traditional software, LLM outputs are non-deterministic, and “correct” is subjective. Because so much judgment is involved, you often need an AI to evaluate an AI, which introduces its own potential failure modes. Making matters more complicated, a single interaction with an LLM can chain retrieval, reasoning, tool calls, and generation, each of which can fail independently.

生成式 AI 打破了许多过去在软件测试中成立的假设。与传统软件不同，LLM 的输出是非确定性的，并且“正确”是主观的。由于涉及大量判断，你通常需要一个 AI 来评估另一个 AI，这引入了其自身潜在的失败模式。更复杂的是，与 LLM 的单次交互可以链接检索、推理、工具调用和生成，其中每一个都可能独立失败。

At Airbnb, we build LLM-powered features across our product, with recent launches including revi...