Watching CI with the GitHub CLI

The pitfall: raw watchers exit early

gh run watch <id> --exit-status and gh pr checks <pr> --watch are NOT safe to arm right after a git push / tag / PR-open:

• Premature exit while QUEUED. Armed immediately after a push, the run has no jobs registered yet. gh sees "nothing in progress" and returns exit 0 — even though the run is still queued/in_progress. (Observed on the v0.1.11 release: the watcher exited 0 while the Test gate job was still running.)
• Transient errors read as failure. A mid-watch HTTP 401: Bad credentials (token refresh) or a 5xx makes the watcher exit non-zero, indistinguishable from a real CI failure. (Also observed on v0.1.11.)
• No native fix. There is no gh run watch flag that waits-for-existence or tolerates transient errors (--interval only tunes the poll cadence). The script below is the fix.

The rule

Never trust a raw watcher's exit code alone. Always re-verify terminal status with gh run view <id> --json status,conclusion (a run is done only when status == "completed") or gh pr view <pr> --json statusCheckRollup (done only when every item is terminal). And always fail-fast — act on the first failing check, never wait for all checks to finish (AGENTS.md fail-fast discipline).

The blessed tool bakes all of this in — prefer it over a hand-rolled watcher.

The blessed tool: scripts/ci-watch.ts

Blocks until the target is truly terminal, then exits with a trustworthy status. Dogfoods nub; runs under plain Node too.

nub  scripts/ci-watch.ts --run <run-id> [--repo o/r] [--timeout <min>]
node scripts/ci-watch.ts --pr  <number> [--repo o/r] [--timeout <min>]

• --run <run-id> — watch a workflow run (polls gh run view --json status,conclusion,jobs).
• --pr <number> — watch a PR's check rollup (polls gh pr view --json statusCheckRollup,…).
• --repo <owner/repo> — defaults to the current repo.
• --timeout <minutes> — wall-clock cap before giving up as pending (default 45).
• --required <names> — comma-separated branch-protection check names to gate on (e.g. --required "CI gate"). Success fires the instant every required check is green; a ghost or a non-required check — pending or failed — never blocks, matching branch-protection semantics. The precise, hang-proof gate for a merge watcher — prefer it when you know the required check name.
• --no-progress <minutes> — how long an unchanged incomplete set (all required/named checks already green, only a ghost left) may sit before exiting 4 STUCK-but-safe (default 8).

What it fixes: waits for the target to EXIST (a not-found / no-jobs-yet target is "keep polling," never "done"); polls authoritative terminal state (status == "completed" / every required/named rollup item terminal+green); fails fast on the first FAILURE/CANCELLED/TIMED_OUT/STARTUP_FAILURE; never hangs on a ghost (see below); tolerates transient gh/API errors (retried with backoff, not treated as a run failure); uses gh's stored token implicitly (high rate limit) with exponential jittered backoff (10s → cap 60s, 90s if unauthenticated).

The #327 ghost — why a strict "all checks terminal" gate hangs

GitHub occasionally registers a check-run that never reports a status: it stays PENDING, nameless, forever. A watcher that waits for every rollup item to be terminal then blocks indefinitely even though every real check is green — PR #327 sat reviewed-and-green for hours this way, its merge watcher parked on one nameless ghost. The fix: a nameless / never-terminating non-required check does not block a green verdict. Once every named check is green and the incomplete set has been unchanged for --no-progress minutes, the watcher exits 4 (STUCK-but-safe) with an actionable summary — the caller --admin merges instead of hanging. A named pending check is never treated as a ghost, so a real in-flight check is never green-lit early.

Exit-code contract

The final stdout line is a single self-describing summary, e.g. CI-WATCH run 27972328590: SUCCESS (25 job(s) green), CI-WATCH pr 73: FAILURE — check "Test (ubuntu-latest, node 22.13)" → FAILURE (https://…), or CI-WATCH pr 327: STUCK — required/named checks GREEN (51), 1 non-terminal ghost/non-required check(s): (unnamed); safe to --admin merge.

Run it detached (CAVEAT: a long wait STRANDS — see the cron-heartbeat section below; this is for SHORT, synchronously-observed gates only)

It's designed to run as a detached run_in_background Bash task that re-invokes the orchestrator on exit — read the outcome from the tail (the CI-WATCH … line) and gate on the exit code:

nub scripts/ci-watch.ts --run "$RUN_ID" --repo nubjs/nub   # run_in_background: true

NEVER pipe the watcher — a pipe silently discards the verdict you gate on

The exit code IS the contract, and a shell pipeline throws it away: in ci-watch.ts … | tail -20, $? is tail's status, which is 0 essentially always, so a FAILED CI reads as SUCCESS. The pipe also buffers stdout until the pipeline ends, so the log stays empty while the watcher runs and there is nothing to eyeball either. This burned a real thread (2026-07-28, PR #604): the watcher ran for minutes, produced no output, and would have reported a red matrix as green.

# WRONG — $? is tail's, a red CI looks green, and no interim output
node scripts/ci-watch.ts --pr 604 --repo nubjs/nub | tail -25; echo "exit=$?"

# RIGHT — redirect, then gate on the watcher's OWN status
node scripts/ci-watch.ts --pr 604 --repo nubjs/nub > /tmp/ci604.log 2>&1; rc=$?
tail -15 /tmp/ci604.log; echo "exit=$rc"

main() warns on stderr when it detects a piped stdout (fd 1 is a FIFO for | cmd, not for > file), but do not rely on noticing the warning — redirect by default. Bash ${PIPESTATUS[0]} / zsh ${pipestatus[1]} also recover the real status if a pipe is unavoidable.

Cross-check the rollup regardless of what the watcher says. Never conclude from watcher output alone that CI is green or that "it's still running, so there's nothing to do" — read gh pr view <pr> --json statusCheckRollup,mergeStateStatus directly and act on any FAILURE (AGENTS.md fail-fast discipline). A watcher that has produced nothing for a long stretch is a suspect, not a status.

For a merge-queue drain, prefer scripts/merge-cascade.ts (it gates positively and merges on green); reach for ci-watch.ts when you just need to block on one run/PR and branch on the result.

Merge-on-green — the ORCHESTRATOR runs the watcher as a background shell; NEVER a watcher sub-agent (learned 2026-06-25)

What stranded EVERY merge in the v0.2.2 floor-fix batch (#163/#164) was NOT background shells and NOT the script — it was dispatching a ci-watch sub-agent. A sub-agent that backgrounds a watch and then rests ORPHANS the command: the sub-agent isn't the orchestrator, so when its background process exits there's nothing wired to re-invoke the orchestrator and merge. Never dispatch a sub-agent to watch CI.

The proven pattern: the ORCHESTRATOR runs the blocking watcher as its OWN run_in_background Bash task. A background Bash command persists ACROSS TURNS and re-invokes the orchestrator when it exits (the Bash tool contract). So node scripts/merge-cascade.ts --max-minutes <n> (drains .fray/merge-queue.jsonl: watch → gate → merge → ff-pull → exit), launched by the orchestrator with run_in_background: true, IS the durable merge mechanism — on exit the orchestrator is re-invoked and reconciles.

Recipe:

1. Enqueue: append {"pr":N,"branch":"…","thread":"…","note":"…"} (optional "hold":true) to .fray/merge-queue.jsonl. Enqueue UNHELD only once the PR's FINAL head is pushed — a stale head can be green-but-wrong, so verify the head/rebase before it's mergeable.
2. Watch: the ORCHESTRATOR runs node scripts/merge-cascade.ts --max-minutes 40 with run_in_background: true. It gates positively on the required CI gate (present + SUCCESS) + mergeable, merges --squash --admin, ff-pulls, dequeues, exits → re-invokes the orchestrator. It shares ci-watch's #327 ghost carve-out (scripts/lib/ci-rollup.ts): a nameless/never-terminating ghost — or any non-required check, pending OR failed — is non-blocking, so the drain can't hang the way #327 stranded a merge; a still-running or failed REQUIRED gate always holds/blocks, so a red PR is never mis-merged.
3. Landing agents PUSH-THEN-EXIT — they never watch; they report pushed <sha>, queued. The orchestrator's background watcher owns merge-on-green.

A CronCreate heartbeat (every ~4 min, one non-blocking gh pr view poll per queued PR, merge-on-green) is a FALLBACK only if a background shell ever proves unreliable — the orchestrator background shell above is the default and what historically worked. Reach for the blocking ci-watch.ts directly only for a single-run gate you observe synchronously.

Self-contained sub-agents — run_in_background works for them TOO; the orchestrator must NOT preempt (corrected 2026-06-26)

There is NO fundamental orchestrator-vs-sub-agent difference for run_in_background: a backgrounded Bash command persists and re-invokes ITS LAUNCHER on exit — orchestrator or sub-agent alike. Proven: a ci-watch sub-agent backgrounded a watch, rested, was re-invoked when CI went terminal, and reported the result. The earlier "a sub-agent watch ORPHANS" claim was a MISDIAGNOSIS — the watchers were working; the orchestrator PREEMPTED them by impatiently checking CI itself and merging manually mid-trace.

The self-contained landing-agent pattern (the goal — one agent traces push→merge):

1. push the branch;
2. launch node scripts/merge-cascade.ts --max-minutes 40 for its OWN PR (or ci-watch.ts) via run_in_background: true;
3. END its turn (rest);
4. it is RE-INVOKED when the command exits → reports merged / red, and iterates (fix-if-red → re-push → re-watch).

DO NOT preempt a landing agent's background watch — let it trace to merge and report. That impatience, not any orphaning, is what broke the flow.

The FOREGROUND ci-watch.ts --chunk loop below is a FALLBACK only — for an agent that must actively iterate in the foreground and can't rest. Run ci-watch.ts in the FOREGROUND in chunks under the Bash cap, looping on pending:

# Bash tool: foreground (NOT run_in_background), timeout: 570000  (9.5 min, under the 600000 cap)
nub scripts/ci-watch.ts --pr <N> --chunk          # --chunk caps the watch ~9 min and exits 2 with "RERUN to continue" if still pending
#   exit 0 = green → act    exit 1 = red → fix + re-push    exit 2 = pending → RE-RUN the SAME command    exit 3 = error

Loop: while it exits 2, call it again — each chunk completes within the cap (no kill, no orphan). The sub-agent blocks in the foreground the whole time, which is FINE: it's backgrounded relative to the ORCHESTRATOR, so the main loop stays responsive.

Dispatch prompts for a self-gating landing agent MUST spell this out — a sub-agent won't infer the foreground-chunk loop. This is ONLY for when the sub-agent needs to SEE its own result to iterate. For the common "push and let it merge" case, the agent push-then-exits and the ORCHESTRATOR's background shell / merge-queue owns merge-on-green (above).