Grab Bench：评估AI在Grab型生产工作中的表现

Introduction

简介

What worried us wasn’t the hallucination, it was the subtle plausibility. Answers an engineer could easily read past and accept: a right-looking Structured Query Language (SQL) query, a plausible tool call, an innocent profile update, or a patch that satisfied the surface tests.

让我们担忧的不是幻觉，而是那种微妙的合理性。工程师很容易一扫而过并接受的回答：一个看起来正确的结构化查询语言（SQL）查询、一个看似合理的工具调用、一个无害的个人资料更新，或者一个通过了表面测试的补丁。

When we analyzed the row-level failures, a clear pattern emerged:

当我们分析行级失败时，一个清晰的模式浮现出来：

• SQL generation: kept the query shape but changed the underlying metric.
• SQL 生成：保持了查询结构，但更改了底层指标。
• Tool calling: selected the right tool family but drifted on parameters.
• 工具调用：选择了正确的工具族，但在参数上发生了偏移。
• Profile updates: cited every event instead of only the evidence that supported the claim.
• 画像更新：引用了每一个事件，而不仅仅是支持该声明的证据。
• Coding agents: passed visible tests while missing a hidden stateful invariant.
• 编码智能体：通过了可见测试，却遗漏了一个隐藏的状态不变量。

Grab Bench bridges this exact gap. Grab Bench is a configurable eval (evaluation) harness for artificial intelligence (AI) systems on Grab-shaped work. It runs model providers through task plugins, records one row per case/model pair, and uses deterministic scorers or large language model (LLM) judges depending on the task. We treat the eval like software: version it, run baselines, keep score records, and make the failure modes visible enough for a team to debug.

Grab Bench 弥合了这一确切的差距。Grab Bench 是一个可配置的评估（evaluation）框架，用于在 Grab 业务形态的工作中运行人工智能（AI）系统。它通过任务插件运行模型提供者，为每个用例/模型对记录一行，并根据任务使用确定性评分器或大语言模型（LLM）评估器。我们像对待软件一样对待评估：对其进行版本控制、运行基线、保留得分记录，并使故障模式足够可见以便团队进行调试。

This write-up focuses on the design choices behind that work.

本文重点介绍该工作背后的设计选择。

The problem: plausible is not correct

问题所在：看似合理并不等于正确

Public leaderboards are still useful; we read them too. They just answer a different question. A product team needs to know whether a model can preserve a metric definition, obey an internal tool contract, stay cautious with weak evidence, or make a code change without breaking behaviour hidden from the prompt.

公开排行榜依然有用，我们也会参考它们，只是它们回答的...