Testing bench_env Task Judges

Overview

A single positive + a single "opposite of positive" negative is not a test — it's a tautology. Real coverage means the negative exercises a failure mode an Agent could realistically produce. Most defects ship because tests stop at one reverse pattern.

Authoritative ref: bench_env/docs/task/TASK_TESTING_GUIDE.md §4 (read §4.3.2 through §4.3.5 before writing cases).

Rule 1 — Positive answer must be natural language (§4.3.2)

AnswerTask positives with bare ground-truth values (answer="32", answer="多云") bypass match_value's fuzzy/number-extraction logic — the test proves nothing.

# ❌ bare ground truth — test passes tautologically
_make_input(state, state, answer="多云")

# ✅ natural language, as an Agent would actually respond
_make_input(state, state, answer="上海今天天气多云转晴")

Format must include ground truth as substring / extractable number, plus realistic surrounding context.

Rule 2 — Pick negative patterns from the taxonomy (§4.3.3)

Blanket answer="错误答案" / curr_state == init_state is a single degenerate pattern. For each task, pick ≥1 pattern from the matching table:

Query tasks

Operate tasks

Cross-app tasks

Every negative case must name a pattern from the relevant table. Complex tasks (multi-field, cross-app) need ≥2 different patterns.

Rule 3 — hybrid tasks need ≥2 negatives (§4.3.1)

hybrid 任务同时检查状态变更和 answer，失败维度独立，≥2 反例：

Rule 4 — Cover match_value boundary cases at suite level (§4.3.4)

These are suite-level concerns; add them as extra cases (name them "TaskName_suffix" and put them in HYBRID_EXTRA_NEGATIVE_CASES or similar to avoid breaking test_*_judge_matrix_complete).

Pick representative tasks and cover:

• 中文数字: at least one positive with answer="北京现在二十度"
• 空 answer: at least one AnswerTask negative with answer=None
• 多数字干扰: GT value in answer while other numbers are also present
• 子串包含陷阱: for any 肯定/否定 query, negative proves "通过" ⊂ "未通过" handled
• 尾零格式: for 小数金额 tasks, positive with "总共278.20元"

Rule 5 — Multi-format positives for structured values (§4.3.5)

When get_answer() returns time ("09:54") or duration ("0小时59分"), match_value substring fails on equivalent formats. If the task uses match_time / match_duration in check_goals, the test must parametrize positives:

@pytest.mark.parametrize("answer", [
    "G7010，1小时10分，13:10",         # exact
    "G7010，70分钟，下午1点10分",       # chinese natural
    "G7010，1小时10分钟，13:10到达",    # mixed format
], ids=["exact", "chinese_natural", "mixed"])
async def test_fastest_train_flexible(self, env, answer): ...

Rule 6 — Keep matrix-complete invariant intact

test_*_judge_matrix_complete requires exactly one main positive + one main negative per task name. Extra variants go into a separate list (e.g. HYBRID_EXTRA_NEGATIVE_CASES) with suffixed names ("CheckBalance_empty_answer").

Rationalization table

Checklist

• Each task has ≥1 positive with natural-language answer
• Each task has ≥1 negative; pattern named from §4.3.3 tables (not mirror-of-positive)
• hybrid tasks have ≥2 negatives covering each independent fail axis
• Suite-level: §4.3.4 boundary cases sampled (中文数字 / 空 answer / 多数字干扰 / 子串陷阱 / 尾零)
• Structured-value tasks (time / duration / date): §4.3.5 parametrized positives
• New cases don't break test_*_judge_matrix_complete (variants go to *_EXTRA_*)
• Uses make_judge_input from conftest.py; no defensive .get() on state
• Ran pytest bench_env/tests/<suite>/ -m "not live" -v and it passes