Run the app-builder benchmark

The benchmark measures cost (exact wire token counts × pinned list prices), quality (60% fixed Playwright customer-journey suites + 25% security probes + 15% single LLM judge, per checkpoint, averaged over three milestones) and duration, for building three apps (Relay CRM, Deskhero, Portalis) with Dyad's local-agent mode. Everything is n=1 per cell, so a single bad line in one milestone moves an app column by 30 points; read every number with that in mind and say so when reporting.

Where things are

BENCH = the benchmarks/app-builder directory. It is not on every checkout: look in the current repo first, then the sibling checkout the user runs benchmarks from (as of 2026-09 that is /Volumes/essd/dyad-2, whose benchmarks/app-builder carries uncommitted hardening). Export BENCH for the bundled scripts if it is not under this repo. Key pieces:

Cell ids are <slug>-<app>[-<effort>][-<label>] where slug = the model spec with the provider stripped and non [a-z0-9.-] chars replaced by _ (openrouter/z-ai/glm-5.3 → z-ai_glm-5.3). The unsuffixed cell is the product default; a tier suffix only exists for forced tiers. Artifacts per cell: results/s-cell/<cell>.summary.json (+ .m<k>.messages.json, checkouts/<cell>), results/s-score/<cell>-ckpt<k>-a1.json, results/judge/<cell>-m<k>.json, proxy/logs/requests-<cell>.jsonl.

Procedure

1. Preconditions. curl -sf http://127.0.0.1:7788/__sim/state must answer (else cd $BENCH/neon-sim && node server.mjs &; if :443 is refused, quit the Tailscale app, which holds it, and restart). DYAD_PRO_KEY must be in the environment or the repo .env. Free ports for the arm (below).

2. New model? Follow references/adding-a-model.md first: pin the live catalog, check maxOutputTokens and defaultEffortLevel, pin list pricing with provenance, register the slug in report.mjs / make-scores.sh. A model missing from the proxy's pin is refused by run-cell for a reason (references/traps.md).

3. Launch an arm in the background and watch its [arm] lines:

   BENCH=... scripts/run-arm.sh openai/gpt-6-astra catalog-2026-09-04.json            # product default
   BENCH=... scripts/run-arm.sh openai/gpt-6-astra catalog-2026-09-04.json --effort low \
       --proxy-base 7795 --block-base 7 --score-port 3100                              # second arm, own ports
   BENCH=... scripts/run-arm.sh openrouter/meta/muse-spark-1.3 catalog-…-musefix.json --sequential
   

   Port plan: arm A uses proxies 7789/7791/7793, port blocks 4/5/6, scoring on 3000; arm B 7795/7797/7799, blocks 7/8/9, 3100. Three parallel builds need ~5 GB; six is the ceiling on the 16 GB mini, so start a second arm after the first's builds finish unless vm_stat shows ≥5 GB free + inactive. Use --sequential when a provider throttles (429/503 or multi-minute request durations in the ledger); parallel cells against a rate-limited upstream double cost and wall-clock.

4. Verify the wire within the first minute — this is what separates a real measurement from a mislabelled one:

   node scripts/verify-ledger.mjs <cell>... --bench $BENCH [--expect-effort low] [--expect-max 65536]
   

   Expect: the intended model string, the intended effort (or thinking:…budget=N for Gemini, none for OpenRouter models, which get no effort field), the catalog's max output tokens (OpenAI-compatible paths may send none — fine unless it is an Anthropic model), all HTTP 200. The gpt-5.6-luna rows are Dyad's own side tasks and are normal.

5. When a build exits. exit=0 → read summary.json: every milestone has a snapshotDb, errorEvents 0, note overSoftCap. exit=1 → grep "Local agent stream error\|AssertionError" build-<app>.log. A stream error aborts the cell before its checkout is archived, so it cannot be scored. Classify it: provider-side (429/503, "Corrupted thought signature", context-length 400 from a bad catalog value) → rerun with --label r2 (after the arm finishes, alone if the provider is throttled) and map it in report.mjs CELL_OVERRIDES with a one-line reason; model-side (it wrote broken code) → score as-is. Quarantine invalid artifacts under results/s-cell/quarantine/ and rename their ledgers rather than deleting; the report must stay reproducible.

6. After scoring. Per checkpoint, results/s-score/<cell>-ckpt<k>-a1.json has buildStatus (ok / build_failed / server_error; harness_error is unscored, not zero) and failures; the Playwright messages are in results/s-score/cuj-<cell>-ckpt<k>-a1.json. Then:

   scripts/find-missing-judges.sh --bench $BENCH [--fix]   # a missing verdict scores as 0
   cd $BENCH && node report.mjs && ./make-scores.sh         # and ./make-gallery.sh if tours were recorded
   

   Confirm the scores page still serves (curl -s -o /dev/null -w "%{http_code}" https://wwwillchen-bot-mini.tail5775e4.ts.net:8443/scores.html).

7. Diagnose before you explain. When an app column is low, find the first failing assertion in the cuj JSON; most collapses are one bug (duplicate data-testid, a workspace switch that 400s, a missing default workspace) that takes every downstream journey with it. If static reading doesn't settle it, reproduce: clone the checkpoint snapshot via POST :7788/__sim/clone, build and serve the checkout, drive the failing journey with a small Playwright script logging /api responses, and inspect the clone DB with psql. Say what the bug is, not "the model did badly".

8. Optional demo video. ./record-tours.sh <cell> <app> <port> "<title>" "<subtitle>" records a zoomed, slow-paced walkthrough of checkpoint 3; add the cell to make-gallery.sh MODELS and rerun ./serve-gallery.sh.

Reporting results

Lead with the table, then the leaderboard position, then the failure story, then the caveats. Always state: per-app composite / cost / build minutes, overall, total cost, where it lands on the headline table, which cells are labelled reruns and why, any harness intervention (a capped catalog value, a raised judge cap), and that n=1 means a few points is noise. Cost comparisons are only meaningful against the same apps; quote list-price ratios alongside measured bills, because fast models routinely bill far below their list-price ratio (fewer turns, less re-read context).

Effort arms: the headline row is the product default (from the pin's defaultEffortLevel at the time of the run — it can change; headlineSuffix in report.mjs handles a flipped default); other tiers go in the sweep table. Never present a forced tier as the default without saying so.

Expectations (for sanity-checking a run)

Read references/traps.md before touching the proxy, catalog, pricing, judge or drain check: each entry is a past run that reported a wrong number.