Aider Polyglot evaluations

Use benchmarks/polyglot/scripts/run_eval.py as the canonical harness. Keep the corpus commit, mini-v2 task list, public prompt construction, timeouts, and upstream scorer unchanged while comparing configurations.

Prepare

1. From the repository root, run:

   ./benchmarks/polyglot/bootstrap.sh
   ./build.sh
   

   Bootstrap clones Aider-AI/polyglot-benchmark into benchmarks/work/polyglot-benchmark, checks all required language toolchains, and checks out the pinned corpus revision.

2. Before spending model calls, run git status --short in both the Capstan repository and corpus. Refuse a dirty corpus. Record a dirty Capstan worktree as non-reproducible exploratory work, not a publishable result.

3. Put provider credentials in the environment, never in a committed file.

4. Confirm an agent's authentication with one neutral request before a full suite. Use fresh output directories outside the corpus.

Canonical comparison

For the published-style local comparison, use direct DeepSeek V4 Pro, medium reasoning, 240-second agent timeout, and sequential full suites:

• Capstan agent command:

  {repo_root}/build/capstan run --benchmark --no-wiki --provider deepseek --model deepseek-v4-pro --profile implement --reasoning-effort medium --max-turns 40 --prompt-file {prompt_file} --workdir {workdir} --workspace {workdir} --json --trace-file {trace_file}
  

• OpenCode agent command:

  python3 {repo_root}/benchmarks/polyglot/scripts/run_opencode.py --prompt-file {prompt_file} --workdir {workdir} --model deepseek/deepseek-v4-pro --variant medium
  

Pass either command as run_eval.py --agent-command. First run the complete harness command with --dry-run; only remove it after checking corpus revision, task matrix, output directory, command, and model. The OpenRouter alternatives are documented separately in benchmarks/polyglot/README.md; do not mix direct DeepSeek and OpenRouter outputs under one --comparison-id.

Run each complete 12-task suite three times in fresh r1, r2, and r3 output directories. Run suites sequentially; do not reuse partial output, selectively rerun failed tasks, or run comparison configurations concurrently. Balance provider load over time while keeping execution sequential:

r1: Capstan -> OpenCode
r2: OpenCode -> Capstan
r3: Capstan -> OpenCode

For every Capstan command in the published isolated comparison, include --trace-file {trace_file}. For both agents, pass the same --replicate-id rN for corresponding repetitions and the same --comparison-id only when provider route, model, reasoning setting, corpus, task matrix, and timeouts are comparable. The documented commands in benchmarks/polyglot/README.md are the source of truth for these flags and placeholders.

Optional trace-free diagnosis

For a separate, non-isolated diagnostic configuration, pass run_eval.py --canonical-log PATH and include --session-id {attempt_id} in its Capstan agent command. Omit --benchmark; --trace-file is not required for this mode. Configure the local capstan.log.v1 lifecycle file exporter externally through normal runtime configuration: the harness only reads PATH, not configures logging. Optional harness --telemetry-context supplies benchmark identity as OTEL_RESOURCE_ATTRIBUTES; it does not enable an exporter.

The harness enforces the session placeholder and rejects --benchmark with both --canonical-log and --telemetry-context: isolated benchmark mode disables native context and export. Non-isolated runs may load normal extension surfaces; record their configuration and keep these diagnostics separate from the published isolated comparison, with a distinct configuration/comparison ID. Public prompts, corpus, task selection, scoring, and fairness rules still apply.

Canonical ingestion selects exactly one trace/run by generated session ID and fails closed on ambiguous/corrupt input, without legacy fallback. It consumes local JSONL only, not OTLP or backend queries; provide a complete ordered input across rotations. Missing files yield missing telemetry; unfinished spans are partial, and paired events cannot prove no spans were dropped.

Report native correlation, root outcome, and producer-supplied root measurements when present: duration/turns, request/tool counts, and model/tool/permission/ subagent/unattributed/overlap time. The analyzer adapts explicit runtime totals; it never infers them from log timestamps or child spans. Old or interrupted records may lack totals: report N/A and reduced metric coverage, not zero. Harness process measurements and upstream scoring remain authoritative; timeout or outcome/exit disagreement makes telemetry inconsistent. See specs/benchmarks.md for the adapter contract.

Evaluation integrity

• Give agents only the public exercise prompt. Never add solution paths, implementation hints, task-specific instructions, or agent-specific help.
• Do not expose .meta or .docs to the agent workspace.
• Score only upstream tests in the clean scoring copy. Agent exit status or self-reported success is not a pass.
• Keep test timeout and task selection fixed. A changed task list, corpus, timeout, model, or harness is a new benchmark configuration.
• Treat agent_timeout, agent_error, test_failure, and harness_error as distinct outcomes.
• If the harness is defective, fix it and invalidate affected runs. Never edit CSV or summary results manually.

Results

After all repetitions finish, run the trace analyzer over every corresponding output directory, for example:

python3 benchmarks/polyglot/scripts/analyze_traces.py \
  --capstan /tmp/capstan-polyglot-r1 \
  --capstan /tmp/capstan-polyglot-r2 \
  --capstan /tmp/capstan-polyglot-r3 \
  --other /tmp/opencode-polyglot-r1 \
  --other /tmp/opencode-polyglot-r2 \
  --other /tmp/opencode-polyglot-r3

Treat process wall time as authoritative for all attempts, including timeouts and crashes. Use model/tool/permission/subagent/unattributed breakdowns only from complete, process-consistent Capstan traces. Preserve .partial traces for diagnosis, but do not include partial, corrupt, or inconsistent telemetry in breakdown averages. For published isolated comparisons, runtime logs are diagnostic context and never replace the per-task legacy trace. The optional canonical diagnostic mode above is not a replacement for published legacy breakdown evidence. Preserve OpenCode's raw event logs; compare normalized metrics without treating format differences as performance differences.

Report score, grouped failure modes, per-task and aggregate wall time, CPU, peak RSS, corpus commit, clean revisions, agent versions, machine details, provider/model/reasoning settings, and paths to raw output. Also report model, tool, permission, subagent, and unattributed time; model request and tool-call counts; trace/metric coverage; and counts of partial, corrupt, and inconsistent traces. State that results are workload- and environment-specific; they are not a universal ranking.

Use benchmarks/polyglot/promptfoo/ only when an HTML report is requested. Promptfoo is an optional orchestration layer; run_eval.py owns task isolation and scoring. Read its README before use and keep a filled local config only at its ignored benchmarks/polyglot/promptfoo/config.local.yaml path.