Triage Failing Experiments and Evaluations

From "the run failed" to the specific rows, evaluators, and inputs responsible, then to a root cause. Work the steps in order.

Prerequisites

Step 1: Find the Failing Run

langwatch experiment list --limit 20 -o json          # experiments in the project
langwatch experiment list-runs <slug> -o json         # runs for one experiment
langwatch experiment status <slug> -o json            # latest run: status, progress, errors
langwatch experiment status <slug> --run-id <id> -o json

A run can fail two ways, and they triage differently:

• Execution failure: the run errored out or stalled. status shows the error; jump to Step 4.
• Score regression: the run completed but evaluator scores dropped or rows failed. Continue to Step 2.

Step 2: Isolate the Failing Rows

langwatch experiment results <slug> --filter failed -o json
langwatch experiment results <slug> --filter failed --evaluator <name> -o json
langwatch experiment results <slug> --run-id <id> --limit 50 -o json

• --filter failed keeps only the rows that failed at least one evaluator. Start there, not with the full result set.
• --evaluator <name> shows one evaluator's column when several ran: is the regression concentrated in one evaluator (a scoring problem) or spread across all of them (a real behavior regression)?

For each failing row, note the input, the expected output (from the dataset), and the actual output. Rows that fail the SAME way point at one root cause; rows that fail differently suggest flakiness or a noisy evaluator.

Step 3: Inspect the Evaluators

langwatch evaluator list -o json
langwatch evaluator get <idOrSlug> -o json

Before blaming the agent, rule out the scorer:

• LLM-judge evaluators: check the model in settings. A judge model that changed, is rate-limited, or is too weak for the rubric produces score swings that have nothing to do with the agent.
• Thresholds: a score of 0.49 vs a pass threshold of 0.5 is a borderline judge, not a regression. Look at the score distribution across rows, not just pass/fail.
• Deterministic evaluators (exact match, JSON validity): these don't drift; failures here are real.

Step 4: Root Cause

1. Compare the failing run against the last passing one: what changed (prompt version, model, dataset, code)? git log on prompts and agent code usually answers this directly.
2. If rows fail on retrieval or context: inspect a production trace of the same path (langwatch trace search / langwatch trace get; see the debug-with-langwatch recipe).
3. If the dataset itself looks wrong (stale expected outputs, bad rows), fix the dataset. Use langwatch dataset get <slugOrId> to inspect it.
4. Apply the fix and re-run:

langwatch experiment run <slug> --wait
langwatch experiment status <slug> -o json

Step 5: Prevent the Recurrence

• If the failure mode wasn't covered by any evaluator, add one (langwatch evaluator create) and wire it into the experiment.
• If it only shows up in production, set up a monitor (langwatch monitor create) so online evaluation catches it before the next experiment does.