从数周到一天：我们如何让 LLM 评估快到足以进行迭代
[
[
The Airbnb Tech Blog
Airbnb 技术博客
](https://medium.com/airbnb-engineering?source=post_page---publication_nav-53c7c27702d5-14e2d35198b4---------------------------------------)
](https://medium.com/airbnb-engineering?source=post_page---publication_nav-53c7c27702d5-14e2d35198b4---------------------------------------)
[
[
](https://medium.com/airbnb-engineering?source=post_page---post_publication_sidebar-53c7c27702d5-14e2d35198b4---------------------------------------)
](https://medium.com/airbnb-engineering?source=post_page---post_publication_sidebar-53c7c27702d5-14e2d35198b4---------------------------------------)
Creative engineers and data scientists building a world where you can belong anywhere. http://airbnb.io
富有创造力的工程师和数据科学家正在构建一个让你在任何地方都能找到归属感的世界。http://airbnb.io
Training an LLM is the easy part. The hard part is designing experiments and evaluations that you can trust enough to know whether the new model is actually an improvement.
训练LLM是简单的部分。困难的部分是设计实验和评估，让你有足够的信心去知道新模型是否真的有所改进。
A person wearing a yellow long-sleeve shirt lines up a shot on a pool table with bright red felt. Their hand forms a bridge for the cue stick, aiming toward several scattered solid and striped billiard balls.
一个穿着黄色长袖衬衫的人正在亮红色台呢的台球桌上瞄准击球。他们的手为球杆搭起手架，瞄准几个散落的纯色和条纹台球。
Introduction
引言
Shipping a production LLM system means iterating fast on improvements to something that is, by construction, non-deterministic. Models drift, judges disagree with themselves, references regenerate as different strings, and bugs may persist until the next release, because retraining takes weeks. Most of this friction comes from infrastructure challenges, not model quality, and the fixes come from classical software engineering techniques.
交付生产级LLM系统意味着对本质上非确定性的事物进行快速迭代改进。模型会发生漂移，评判器会自相矛盾，参考文本会重新生成为不同的字符串，而且由于重新训练需要数周时间，bug可能会一直存在到下一个版本。这种摩擦大部分来自于基础设施挑战，而非模型质量，修复方法则来自于经典的软件工程技术。
At Airbnb, we built reliable LLM infrastructure by addressing four layers. Three correspond to engineering enhancements we’...