Triage Agent Eval Failures

Diagnose a failing scenario in libs/agent-evals and produce a verdict: is the failure real (the playbook under test regressed) or is the test wrong (grader / tape / scenario / judge), or is it just flaky (model non-determinism)?

The thing under test is the playbook doc (packages/shared/docs/agent-onboarding.md), injected as the agent system prompt. Everything else (graders.ts, catalog.ts, scenario.ts, judge prompts) is test scaffolding. Never fix the playbook to satisfy a broken grader, and never loosen a grader to hide a real playbook regression.

Rule 0: rule out flakiness before changing anything

Scenarios run a live model concurrently, so one red run is one sample, not a verdict. Re-run the single failing scenario 3–5× first:

pnpm --filter @novu/agent-evals exec vitest run --config vitest.evals.config.ts -t <scenario-id>

• Fails every run → deterministic failure, continue triage.
• Fails intermittently → flaky. The cause is usually a non-deterministic judge grader or an over-strict regex. Do not edit the playbook. Tighten the grader/judge prompt or accept variance; consider pass@k rather than single-run gating.

To reproduce judge graders locally:

pnpm --filter @novu/agent-evals exec vitest run --config vitest.evals.config.ts -t <scenario-id>

Step 1: identify which grader failed and its kind

Each scenario registers graders in scenarios/<id>/graders.ts. The kind is the strongest triage signal:

• Deterministic graders (catalog.*, contains, matches) inspect the structured RunResult. A fail means the agent's actions/output objectively did not match — or the check is too strict.
• Judge graders (sharedJudgeGraders, judge(...)) call a second LLM pass. A fail is fuzzy and can be the judge prompt's fault, not the agent's.

Find the grader's logic:

Step 2: read the RunResult evidence

Graders read fields off RunResult (src/core/types.ts). Map the failing grader to the field it checks and compare against what the agent actually did in the run output:

• trackedCommands — raw connect command strings (flag checks like --keyless, --secret-key, --slack-config-token).
• toolCalls — every Bash / BashOutput / AskUserQuestion / Read call with args (run_in_background, file_path, picker selectedId).
• polledShellIds / killedShellIds — background-polling and kill behavior.
• capturedUrls / openedFiles — surfaced URLs and opened files (e.g. QR .png, auth-url file).
• finalText / assistantMessages — user-facing report (transcriptText joins these).
• metadata.description — the drafted agent description (persona / infra-token graders).

Step 3: classify the failure

Walk top-down and stop at the first match:

A scenario passes only when every active grader averages ≥ 0.8 (JUDGE_THRESHOLD). A judge returning UNKNOWN becomes skip and scores 1 — it never causes a fail, so an UNKNOWN is not evidence of a real regression.

Step 4: apply one bounded fix, then verify

1. Change only the layer the verdict points to — playbook or test, never both to chase green.
2. Re-run the single scenario (Step 0 command).
3. Confirm the fix holds across the 3–5 re-runs and that no other scenario regressed.
4. If editing a deterministic grader, also run the synthetic unit tests so you don't break grader contracts:

pnpm --filter @novu/agent-evals test

Output format

Report the verdict concisely with cited evidence:

Scenario: <id>
Failing grader: <name> (deterministic | judge)
Re-run result: <N/M failed> → real | flaky
Evidence: <RunResult field + actual vs expected>
Verdict: real playbook regression | test bug (<grader|tape|scenario|judge>) | flaky
Fix target: <file path>  (or: no change — flaky/UNKNOWN)

Additional resources

For worked triage examples (real regression vs test bug vs flaky judge), see reference.md.