investigate-flaky-test

Overview

Investigate a GitHub issue created by detect-flaky-ci and labeled flaky/confirmed: reproduce the non-determinism, find the root cause, classify it, and — if approved — fix it and open a PR.

This mirrors investigate-issue's shape (confidence-gated stop points, phase/* label lifecycle, branch/PR conventions) so both skills feel like the same tool. It is a separate skill, not a mode of investigate-issue, because the actual investigation content is unrelated: no reported version to check, no browser reproduction of user-facing steps, and the fix taxonomy (test-side vs. product-side vs. quarantine) does not apply to ordinary bug reports.

Supports the same two execution modes as investigate-issue:

• interactive (default): stop at every decision gate
• autonomous (--auto, or invoked from /flaky-ci-routine): cross a gate automatically at HIGH confidence, stop at MEDIUM or LOW

Input

$ARGUMENTS is an issue URL or number, optionally with --auto. Parse the issue number and mode the same way investigate-issue does.

Precondition: the issue must carry flaky/confirmed or flaky/suspected (fetch exact label names with gh api repos/growilabs/growi/labels --paginate -q '.[].name' before comparing — never hardcode, and use REST here, not gh label list --json, which a cloud routine's proxy-restricted gh session rejects — see Error Handling). If it is still flaky/observing, stop and report that detect-flaky-ci has not gathered enough evidence yet; do not attempt to lower the bar by investigating early.

These two accepted labels mean different things and change Step 2:

• flaky/confirmed — already empirically proven (Playwright's in-run retry already IS the proof, or a prior vitest threshold-accumulation already reached two independent observations). No confirmation measurement is owed here before proceeding to root-cause work — go straight into Step 2's existing evidence-gathering.
• flaky/suspected — detect-flaky-ci's cheap mechanical mining (diff/ PR mismatch, sandwich pattern, or matrix divergence) found this, but nothing has actually reproduced it live yet. This skill owes it exactly one confirmation measurement — a single push of the repro workflow, which replays the spec 3 times on current master (not the deeper tally described in Step 2's second tier, which is a separate budget for a different purpose) — before treating the flakiness itself as real. See "Step 2, flaky/suspected path" below.

Confidence Framework

Same three levels and same autonomous/interactive behavior as investigate-issue (HIGH → proceed autonomously and state the evidence, MEDIUM/LOW → stop and ask with a recommendation). Applied at two gates in this skill: the fix-classification gate (Step 4) and the PR gate (Step 6). What a stop looks like is fixed for both — see Pausing for a human decision below.

Every comment this skill posts is signed

Each issue comment this skill posts ends with this line, verbatim, underscores included:

_Generated by [Claude Code](https://claude.ai/code)_

It applies to every comment named in this file. Where the line sits differs in one case:

• Ends with the signature — the confirmation tally and the "possible genuine regression" finding (2-E), the deterministic-cause close (Step 3).
• Pause comments (2-E's third outcome, Step 4, 6-B, the Error Handling stops) — the signature sits immediately above the comment's final - Recommendation: line, because the dashboard reads that line off the end of the comment. See Pausing for a human decision for the full shape.
• No signature at all — the **Fix PR**: {URL} marker in 6-C, the one exception, which stays the single fixed line the Dashboard Updater matches, with nothing appended to it.

(### Repro result comments are posted by the repro workflow, not by this skill, and carry no signature of ours.)

Why it is mandatory rather than decoration: flaky-ci-routine.md Step 2-B decides whether a comment came from a human — and therefore whether a paused issue is resumed — by the comment's author type or this signature. Its Shared constants → Automated-author signatures block is the single definition site for both checks and for the exact strings; read it there rather than re-deriving the rule here, and spell the line the same way. The author check cannot carry the decision alone: when the routine runs from a human maintainer's gh token rather than as an app, every comment this skill posts is authored by that human's login with user.type == "User". An unsigned pause comment is then indistinguishable from the human's own answer, and the next run resumes the investigation against its own words.

Pausing for a human decision

Every place this skill stops in autonomous mode without finishing — a MEDIUM or LOW reading at a gate, or a "confirmation not measured" outcome — ends the same way. Do this; the individual stop points below only point here rather than restating it.

1. Add the label flaky/needs-decision (defined in the same Shared constants block) — before posting the comment, not after:

   gh api "repos/growilabs/growi/issues/${ISSUE_NUMBER}/labels" -X POST -f 'labels[]=flaky/needs-decision'
   

   POST is additive, which is what this needs: the tier label (flaky/confirmed / flaky/suspected) and the phase label stay untouched, because being paused is a separate axis from either. Leave the phase label at {EXACT_PHASE_UNDER_INVESTIGATION_LABEL} — the investigation is paused, not resolved.

   Label first, then comment — this order is a contract, not a preference (defined in flaky-ci-routine.md Shared constants → Pause ordering). Labelling first is what puts the pause comment inside the window the dashboard reads the Recommendation from; that block holds the window itself and what goes wrong under the reverse order. Follow it rather than re-deriving it here.

2. Post one comment on the tracking issue saying what was measured, what is missing, and which options are open.

   Before composing the comment, fetch this issue's occurrence frequency — the same count the Dashboard shows as its "First seen / Last seen / Occurrences" columns — and add it as one line in the comment body, above both of the fixed lines described below:

   node bin/flaky-ci/scripts/occurrence-summary.ts --issue "$ISSUE_NUMBER"
   

   On exit 0 this prints one line of JSON with firstSeen, lastSeen (each an ISO-8601 UTC timestamp, or null if no observation date could be parsed) and occurrences (a number). Render it as a single comment line, e.g.:

   - Occurrences: 5 (first seen 2026-09-10T03:00:00Z, last seen 2026-09-23T11:00:00Z)
   

   Where firstSeen or lastSeen is null, write unknown in its place instead of omitting the line — occurrences is always present as a number.

   If this script call fails (non-zero exit, or no output) — e.g. the issue or its comments could not be fetched — omit the frequency line entirely and post the comment as before. Do not retry and do not let this step block or delay posting the pause comment.

   Its last two lines are, in this order — and the frequency line, when present, always sits above both of them, never between them:

   _Generated by [Claude Code](https://claude.ai/code)_
   - Recommendation: <one line>
   

   The - Recommendation: line is the comment's last line — the fixed format flaky-ci-routine.md defines under Shared constants → Recommendation line, which the dashboard reads back verbatim by taking the comment's last non-empty line. The signature therefore sits immediately above it: this pause comment is the one place in this skill where the signature is not the final line, and nothing is lost by that, because the routine matches the signature as a substring anywhere in the body. One line, one option, no wrapping.

   Open the comment with a bold lead-in, never a ### heading, for the reason 2-E gives.

3. Stop there. Nothing is watched afterwards and nothing is scheduled — see "Routine discipline" just before Error Handling. The next routine run picks the issue up once a human has answered.

In interactive mode a human is already in the session: present the same reading and the same one-line recommendation, and ask directly. No label and no comment are required — flaky/needs-decision marks an unattended run that ended paused, so applying it to an issue somebody is actively looking at would put a row on the dashboard's awaiting-decision list with nobody waiting on it.

The stop points this covers. Every one of them pauses exactly as above — label, comment (signed, - Recommendation: last), stop:

Two endings in Error Handling are deliberately not pauses and take no label: the precondition reject (the issue carries neither flaky/confirmed nor flaky/suspected — nothing was investigated, so there is nothing for a human to decide here), and the devcontainer-reproduction fallback (which by its own rule must not lower confidence at all). A deterministic-cause close (Step 3) is not a pause either — it is a conclusion, and it closes the issue.

Resuming after a human decision

flaky-ci-routine.md Step 2-B re-selects a paused issue once a human has answered, and appends one line to this skill's invocation:

Human decision (issue comment {COMMENT_HTML_URL}): {FULL_COMMENT_BODY}

When that line is present:

1. Remove flaky/needs-decision first, before any other work — the routine deliberately leaves the label in place so that a resume which dies early stays visibly paused:

   gh api "repos/growilabs/growi/issues/${ISSUE_NUMBER}/labels/flaky%2Fneeds-decision" -X DELETE
   

2. Treat the quoted body as the answer to the gate this investigation stopped at — Step 4's fix-approach gate or 6-B's PR gate, whichever the pause comment named. The decision it supplies is what crosses that gate; do not re-derive confidence from the evidence and stop again at the same place. The answer can pick any of: proceed with the best-guess fix, quarantine (Step 4's guardrail still applies to how it is done), escalate for human review of the product code, close as unable to reproduce, or close as a deterministic cause (Step 3's close path).

   The 2-A vault pause is narrower, because no measurement is possible there at all: its answer is either "proceed from the logs alone" — go on to Step 3/4 with the missing tally stated as the reason confidence is capped — or "close / park it".

3. Record which option the human chose and that it came from that comment. It goes into whichever comment the resumed path posts first: the "the investigation is continuing, on this basis" comment when the investigation goes on, or the close comment itself when the answer is to close — ### Closed: deterministic cause, not flaky for a deterministic cause (Step 3's template, {DECISION_RECORD} filled in), or, for "close as unable to reproduce", this comment — posted with the same gh api "repos/growilabs/growi/issues/{ISSUE_NUMBER}/comments" -X POST -f body="$(cat <<'EOF' … EOF)" form 2-E uses, and only then gh api "repos/growilabs/growi/issues/{ISSUE_NUMBER}" -X PATCH -f state=closed -f state_reason=not_planned, labels untouched:

   ### Auto-closed: not reproduced within {DAYS_SINCE_NEWEST} days
   
   - Newest observation: {NEWEST}
   - Closed by: `investigate-flaky-test`, on the human decision in {COMMENT_HTML_URL}
   
   {DECISION_RECORD}
   
   Closing as `not planned`: the identity has not recurred since the newest
   observation, and the maintainer decided not to pursue it further. If it
   comes back, `detect-flaky-ci`'s closed-issue path reopens it.
   
   _Generated by [Claude Code](https://claude.ai/code)_
   

   {NEWEST} is the newest observation timestamp among the blocks Step 1 collected (the body's first observation and every ### Additional observation / ### Backfilled observation comment — the same value the routine's 4-B computes), and {DAYS_SINCE_NEWEST} is the whole number of days from it to now. The routine's --stale-days is not available here and no threshold was applied, so neither appears. Only the heading is shared with the routine's 4-E, and that is deliberate: ### Auto-closed: not reproduced within is what the routine's manual-reopen guard (4-C) and its observation-count exclusion (4-B, Step 5) recognize, so a human-decided not-reproduced close behaves like an automatic one if the identity ever comes back. If {NEWEST} cannot be determined, do not post and do not close — pause again per Pausing for a human decision saying so. A close answer never produces a "continuing" comment — the close comment is the only one posted — so writing the record only into a "continuing" comment would drop it entirely on exactly the two answers that end the issue.

   Write it as one paragraph naming the option the human chose and linking their comment URL ({COMMENT_HTML_URL} from the invocation line). When a PR is opened, the PR body's Root Cause section carries the same paragraph as well — the issue comment is not a substitute for it. A gate crossed on a human's say-so must not later read as one the evidence crossed on its own.

If the human's answer does not actually settle the gate (it asks a question, or answers something else), pause again per Pausing for a human decision, saying what is still open. Do not guess at what was meant.

Step 1: Fetch Issue Evidence

Use REST (gh api), not gh issue view --json — the latter is GraphQL-backed and rejected by a cloud routine's proxy-restricted gh session (see Error Handling):

gh api repos/growilabs/growi/issues/{ISSUE_NUMBER}
gh api repos/growilabs/growi/issues/{ISSUE_NUMBER}/comments --paginate

The first call gives title, body, labels[].name, html_url, state; the second gives every comment (each observation detect-flaky-ci appended after the first).

The title is flaky: {IDENTITY_KEY} (set by detect-flaky-ci). Parse it with:

node bin/flaky-ci/scripts/parse-identity-key.ts --title "$TITLE"

This replaces the regex and 4-step split this section used to spell out — see bin/flaky-ci/README.md's contract table for the exact fields (kind, browser, specPath, testTitle, shape). The script never fails (exit 2 is only a missing --title): a job-level fallback or a malformed key is itself the fact being reported, not an error. What to do with each of the three shape values is the judgment that stays here:

• shape: "precise" — a vitest:{SPEC_PATH}:{TEST_TITLE} or playwright:{BROWSER}:{SPEC_PATH}:{TEST_TITLE} key. {BROWSER} is chromium / firefox / webkit and, for Playwright, is normally always present: the same spec can be flaky in one engine and not another, so detect-flaky-ci Step 3 makes it part of the key. Proceed straight to Step 2 reproduction using specPath / testTitle.
• shape: "playwright-job-level" (kind is always "playwright", browser holds the fallback's {BROWSER}, specPath/testTitle are null) — detect-flaky-ci could not isolate which spec was flaky from the CI log alone. The issue body's evidence section will say so explicitly. Do not guess a spec — read the linked run's full Playwright report first (the run URL in the "First observation" section) to find the actual flaky spec before attempting Step 2 reproduction. If the report is no longer available (artifact retention expired), report LOW confidence at Step 4 rather than guessing.
• shape: "malformed" — no detect-flaky-ci path produces one. Stop and report it as a precondition failure rather than guessing a path.

Collect every observation block from the body and comments (run URLs, commits, log excerpts) — later observations may show the failure mode drifting or repeating identically, which is itself evidence for Step 3.

Mark as under investigation, same as investigate-issue. The phase labels begin with an emoji, so this is the read-then-whole-array PATCH form 6-C uses (never gh issue edit, which is GraphQL-backed and rejected in a cloud routine's session; never a DELETE /labels/{name} with an emoji in the path):

LABELS_FILE="${TMPDIR:-/tmp}/flaky-phase-labels-{ISSUE_NUMBER}.json"

gh api "repos/growilabs/growi/issues/{ISSUE_NUMBER}" -q '[.labels[].name]' \
  | jq -c --arg rm '{EXACT_PHASE_NEW_LABEL}' \
          --arg add '{EXACT_PHASE_UNDER_INVESTIGATION_LABEL}' \
          '{labels: ((map(select(. != $rm)) + [$add]) | unique)}' > "$LABELS_FILE"

gh api "repos/growilabs/growi/issues/{ISSUE_NUMBER}" -X PATCH --input "$LABELS_FILE"

Fetch the exact label names first (gh api repos/growilabs/growi/labels --paginate -q '.[].name'), as the Precondition requires; the PATCH replaces the whole set, so it must be computed from the labels read immediately before it.

On a resume, make this edit conditional: if the issue still carries {EXACT_PHASE_NEW_LABEL} — the overlap flaky-ci-routine.md Step 2-C describes, where an issue selected both as newly detected and as human-answered is processed once, as the latter — swap it to {EXACT_PHASE_UNDER_INVESTIGATION_LABEL} as above. Otherwise it is already under investigation and there is nothing to do here. Either way, follow Resuming after a human decision as well.

Step 2: Gather Evidence

flaky/suspected path: take the confirmation measurement first, in parallel

If the issue's label is flaky/suspected (see Precondition), start its one-time confirmation measurement before doing anything else in this step, so it runs in the background while the static analysis below reads.

The measurement is taken by .github/workflows/flaky-repro.yml: a push to a flaky-repro/** branch whose head commit carries a Flaky-Repro-* git trailer block makes that workflow replay one spec N times on real CI infrastructure (the same toolchain, MongoDB replica set and Elasticsearch ci-app.yml uses) and post the tally to the tracking issue. Driving it needs only the two abilities this skill always has — pushing a branch and reading REST. (gh run rerun is not used at this gate any more: a cloud routine's token has no actions:write, so that path returned 403 and stopped every flaky/suspected issue before anything was measured.)

This is a hard gate, not an optional courtesy — Step 3 may not start until it is satisfied. Before reaching Step 3, write down (in your own working notes, and later in the issue comment or PR body) one of exactly three outcomes, decided in 2-E below: confirmed, possible genuine regression, or confirmation not measured. Reaching Step 3 without one of these three recorded is the failure mode this gate exists to catch — a plausible-looking root cause already visible in the issue's existing evidence (e.g. matrix divergence) is not a substitute for the measurement, no matter how convincing that existing evidence looks.

2-A: derive the repro request from the identity key

The identity key parsed in Step 1 is vitest:{SPEC_PATH}:{TEST_TITLE}, and SPEC_PATH is already relative to apps/app (e.g. src/server/routes/apiv3/g2g-transfer-key-keep-alive.integ.ts). Pass it verbatim — the workflow rejects a repository-relative apps/app/... value. Derive the vitest project from the same path:

A path matching none of those four suffixes has no repro project at all. The match is on the full suffix, not on .ts alone: a shared setup-hook identity such as test/setup/migrate-mongo.ts (the live case is #11752) ends in a bare .ts and therefore matches no row — it is not app-unit. There is no project to name, so the measurement cannot even be requested. Do not push a request for such an identity, and do not guess a project. Record "confirmation not measured (no repro project for this path)" and pause per Pausing for a human decision, exactly as in 2-E's third outcome — except that 2-F does not apply here (no branch to clean up, because nothing was ever pushed).

A second gap: a spec under src/features/growi-vault/__tests__/ belongs to the app-integration-vault project, which the workflow's allowlist does not accept — and app-integration explicitly excludes those files, so requesting it yields No test files found, a failed job, and no measurement. Do not push a request for such a spec. Record "confirmation not measured (no repro project for vault specs)" and pause per Pausing for a human decision, exactly as in 2-E's third outcome — except that 2-F does not apply here (no branch to clean up on the vault path, because nothing was ever pushed).

A playwright: identity never enters this gate: Playwright's in-run retry is already the proof of non-determinism, so those issues arrive as flaky/confirmed. If one somehow carries flaky/suspected, skip straight to the primary tier below and state in Step 4 that no repro measurement applies to a Playwright identity.

2-B: push the request

ISSUE_NUMBER=11823
SPEC_PATH='src/server/routes/apiv3/g2g-transfer-key-keep-alive.integ.ts'
PROJECT='app-integration'
BRANCH="flaky-repro/issue-${ISSUE_NUMBER}-keep-alive"   # short slug, [a-z0-9-]

git fetch origin master
git checkout -b "$BRANCH" origin/master

# ONE -m. git reads only the LAST paragraph of the message as the trailer
# block, so splitting these lines across several -m flags makes every trailer
# but the last paragraph invisible to the workflow (the one-paragraph rule is
# the Repro Request / Result contract in .kiro/specs/ci-flaky-test-detection's
# design.md).
git commit --allow-empty -m "$(printf 'chore: request a flaky repro for #%s\n\nFlaky-Repro-Spec: %s\nFlaky-Repro-Project: %s\nFlaky-Repro-Mode: file\nFlaky-Repro-Repeat: 3\nFlaky-Repro-Issue: %s\nCo-Authored-By: %s\nClaude-Session: %s\n' \
  "$ISSUE_NUMBER" "$SPEC_PATH" "$PROJECT" "$ISSUE_NUMBER" \
  '{Co-Authored-By value for this session}' '{Claude-Session URL for this session}')"

# Confirm git sees all of them as one trailer block BEFORE pushing:
git log -1 --format='%(trailers:only=true)'

This session's own signature lines go inside the same final paragraph as the five Flaky-Repro-* lines. Commits made from Claude Code end with Co-Authored-By: and Claude-Session: trailers, and they must sit alongside those five — never as a paragraph of their own after them. A blank line between the two groups makes the signature the last paragraph, so git reports only Co-Authored-By / Claude-Session as trailers, the workflow sees no Flaky-Repro-Spec, and the push measures nothing. Use the exact signature lines this session was given, in place of the placeholders above.

That is why the block above stops at the check instead of running the push: the check is a gate, not a printout. It must list all five Flaky-Repro-* lines. If it prints only the signature lines, the paragraph split happened — amend the message and run it again. Push only once the five lines appear (this block continues the same shell as the one above — $BRANCH is still set there):

git push -u origin "$BRANCH"
REPRO_SHA=$(git rev-parse HEAD)

# Hand both values to 2-C, which usually runs in a later shell (see 2-C).
REQUEST_ENV="${TMPDIR:-/tmp}/flaky-repro-request-${ISSUE_NUMBER}.env"
printf 'BRANCH=%s\nREPRO_SHA=%s\n' "$BRANCH" "$REPRO_SHA" > "$REQUEST_ENV"

Flaky-Repro-Mode: file and Flaky-Repro-Repeat: 3 are the defaults this gate uses: 3 independent runs of the one spec, each in its own process. The issue already documents at least one CI failure, so the only thing this measurement has to establish is whether the spec can pass at all on unchanged code — three runs separate that from a real regression, which fails 3/3.

Why the branch is cut from origin/master, not from the failing run's commit. The question this gate answers is "is this spec non-deterministic on current master". The failing commit cannot serve as the base anyway: .github/workflows/flaky-repro.yml is not in its tree, and a workflow only runs if its file exists on the pushed ref, so that push would start nothing at all. If the spec no longer exists on master, the workflow rejects the request (Flaky-Repro-Spec does not exist in this commit) and its check-run is failure, which routes the issue to "confirmation not measured" in 2-E. That is the correct outcome, not a bug — a spec that is gone from master has nothing left to measure.

No normal CI run is created by this push (Requirement 6.7 of the ci-flaky-test-detection spec): the commit is empty, and ci-app.yml / ci-app-prod.yml do not run on flaky-repro/**.

2-C: wait for the flaky-repro check-run

Do not wait here — 2-B is the last thing done before the static analysis below. Go read that now; 2-C through 2-F then run at the end of this step, right before Step 3, while the measurement has had time to finish.

$BRANCH and $REPRO_SHA live only in the shell that 2-B set them in, so that interleave usually means 2-C..2-F run in a different shell with both unset — and an unset $REPRO_SHA turns the poll below into a request for commits//check-runs, which never reports the measurement. Re-establish them from the hand-off file 2-B wrote rather than re-pushing — the branch is already on origin, and pushing a second request would measure the wrong commit:

REQUEST_ENV="${TMPDIR:-/tmp}/flaky-repro-request-${ISSUE_NUMBER}.env"
if [ -r "$REQUEST_ENV" ]; then
  . "$REQUEST_ENV"           # sets BRANCH and REPRO_SHA exactly as 2-B pushed them
else
  echo "no hand-off file for #${ISSUE_NUMBER} — the request cannot be identified"
fi
[ -n "${BRANCH:-}" ] && [ -n "${REPRO_SHA:-}" ] || echo "BRANCH/REPRO_SHA unknown"

Read the values back from that file, not from git ls-remote: the remote lists refs in name order, so a pattern match on flaky-repro/issue-${ISSUE_NUMBER}-* would return the alphabetically first branch, and if an earlier run's branch survived a failed 2-F cleanup that would silently be the old request — and 2-D would then pin its - Commit: match to the old measurement. If the file is missing or either value is empty, do not guess and do not re-push: treat the measurement as confirmation not measured (2-E's third outcome), pause per Pausing for a human decision, and leave any flaky-repro/issue-${ISSUE_NUMBER}-* branches on origin for the human to inspect.

When you come back, call check-runs-facts.ts every 60 seconds, for at most 30 minutes from the push (see bin/flaky-ci/README.md for the full output contract):

CHECKS_JSON="${TMPDIR:-/tmp}/flaky-repro-checks-${ISSUE_NUMBER}.json"
STARTED=$(date +%s)
STARTUP_GRACE=$(( STARTED + 5 * 60 ))
DEADLINE=$(( STARTED + 30 * 60 ))

while :; do
  node bin/flaky-ci/scripts/check-runs-facts.ts --sha "$REPRO_SHA" > "$CHECKS_JSON"
  repro_status=$(jq -r '.flakyRepro.status' "$CHECKS_JSON")
  now=$(date +%s)

  [ "$repro_status" = 'completed' ] && break
  if [ "$repro_status" = 'absent' ] && [ "$now" -ge "$STARTUP_GRACE" ]; then
    echo 'no flaky-repro check-run within 5 minutes — not measured'
    break
  fi
  if [ "$now" -ge "$DEADLINE" ]; then
    echo 'wait capped at 30 minutes'
    break
  fi
  sleep 60
done

Three ways out of the loop:

• .flakyRepro.status reads "completed" → take .flakyRepro.conclusion (jq -r '.flakyRepro.conclusion' "$CHECKS_JSON") to 2-D;
• .flakyRepro.status stays "absent" for the first ~5 minutes — no check-run named flaky-repro has appeared yet — → the instrument never started. Stop polling now rather than burning the full 30 minutes, and treat it as "confirmation not measured" (2-E), naming the likely causes in the stop comment: the branch name did not match flaky-repro/**, or the base commit predates flaky-repro.yml;
• 30 minutes elapse with .flakyRepro.status still "queued" / "in_progress" → "confirmation not measured" as well.

The check-run's conclusion answers only "could a measurement be taken at all": success means the request was valid and the runs happened (the tests themselves may well have failed — that is the measurement), failure means the request was rejected or the setup broke and nothing was measured.

2-D: read the tally from the ### Repro result comment

REPRO_RESULT_JSON="${TMPDIR:-/tmp}/flaky-repro-${ISSUE_NUMBER}.json"

node bin/flaky-ci/scripts/read-repro-result.ts \
  --issue "$ISSUE_NUMBER" --sha "$REPRO_SHA" > "$REPRO_RESULT_JSON"
REPRO_RESULT_STATUS=$?

• Exit 0: $REPRO_RESULT_JSON holds one line of JSON with runs, failed, perRun, workflowRunUrl and commentUrl (see bin/flaky-ci/README.md's contract table). Read the two values 2-E needs: runs=$(jq -r '.runs' "$REPRO_RESULT_JSON") and failed=$(jq -r '.failed' "$REPRO_RESULT_JSON").
• Exit 2: no ### Repro result comment on the issue carries - Commit: ${REPRO_SHA} — treat as "confirmation not measured" (2-E's third outcome).

The comment must be the one this push produced. A tracking issue accumulates ### Repro result comments — the confirmation measurement, a later rate measurement (second tier), the fix verification in Step 6 — so "the newest one" is not the same thing as "mine". The script matches on the - Commit: ${REPRO_SHA} line, not recency, and only when two comments name the same commit (a manually re-run job) does it fall back to picking the newer of the two.

A success conclusion on its own is not evidence that anything was measured. The workflow posts the comment in a separate step that deliberately does not fail the job, so a comment that never arrived (a locked issue, a momentary GitHub outage) leaves the check-run green with only a warning annotation behind it. Require both: the check-run completed success, and read-repro-result (2-D) exits 0 for ${REPRO_SHA} with a runs value of at least 1. If either is missing, the outcome is "confirmation not measured" — the job summary is for human readers, and the issue's existing static evidence is not a substitute.

2-E: decide, from the tally alone

confirmed — a spec that passes at least once on unchanged code, with a recorded CI failure behind it, is non-deterministic. Escalate the label (REST, per Error Handling — the gh issue edit / gh label paths are GraphQL-backed and a cloud routine's session rejects them):

gh api "repos/growilabs/growi/issues/${ISSUE_NUMBER}/labels" -X POST -f 'labels[]=flaky/confirmed'
gh api "repos/growilabs/growi/issues/${ISSUE_NUMBER}/labels/flaky%2Fsuspected" -X DELETE

Then record the tally in a comment, counting the CI failure the issue already documents as one more failing sample — written as recorded CI failure 1 + Failed / Runs + 1. With - Runs: 3 and - Failed: 1 that reads 2 / 4. The decision above used the workflow's own Failed/Runs; the +1 figure is for the record only, never fed back into the comparison.

gh api "repos/growilabs/growi/issues/${ISSUE_NUMBER}/comments" -X POST -f body="$(cat <<'EOF'
**Confirmation measurement** — the repro workflow ran this spec 3 times on
current `master` with no code change: 1 failed, 2 passed
({workflow run URL}). Counting the CI failure already recorded above as a
failing sample, the tally is **2 / 4**.

Escalated `flaky/suspected` → `flaky/confirmed`.

_Generated by [Claude Code](https://claude.ai/code)_
EOF
)"

Use a bold lead-in, not a ### heading: ### Repro result belongs to the workflow, and ### Additional observation / ### Backfilled observation are what detect-flaky-ci counts as occurrences. This comment must not be mistaken for either. This comment ends with the signature; a pause comment is the case where it does not, because a - Recommendation: line follows it — see Every comment this skill posts is signed.

possible genuine regression — every measured run failed on unchanged master code, so "the mining hit was right and this is flaky" is no longer the leading reading. Do not promote the label; leave flaky/suspected in place. Write the finding to the issue (same comment shape and signature, quoting - Runs: and - Failed: and the workflow run URL) stating that the spec failed on every measured run and that a real regression is the leading hypothesis, then carry MEDIUM into Step 4 — see its confidence table.

confirmation not measured — nothing was measured, so this gate cannot be crossed. Write which of the conditions above applied, the check-run URL if there is one, and the likely cause; then pause per Pausing for a human decision (that block owns the comment's last two lines, the flaky/needs-decision label, and the "stop, watch nothing" part), and clean up the branch (2-F) on the way out.

Do not let the issue's existing static evidence stand in for the missing measurement, and do not fold this into any later tally: a measurement that did not happen is not evidence of anything.

2-F: delete the confirmation branch

git push origin --delete "$BRANCH"
git switch -          # leave the throwaway ref before anything branches again
git branch -D "$BRANCH"
rm -f "$REPRO_RESULT_JSON"

Do this on every exit path — including the ones where no result was ever read (rejected request, timeout, check-run absent), or flaky-repro/** refs accumulate on the remote. Deleting the branch starts no workflow run (task 1.6 measured this: the flaky-repro run count was unchanged after its three self-test branches were deleted).

Leaving the local branch is not cosmetic: every later step branches from wherever HEAD is, so staying on the repro branch would make Step 5-A cut the fix branch off it and carry the empty chore: request a flaky repro commit into the fix PR.

Primary tier — CI log analysis (always available, no live services needed)

This skill is expected to run unattended, including from a cloud routine whose checkout has no MongoDB replica set, no Elasticsearch, and no browsers. Do not make CI-log analysis a fallback for when reproduction "isn't available" — treat it as the normal, primary path, and treat live reproduction (below) as a bonus when the current environment happens to support it.

Pull more history than what's already in the issue, using the same tools detect-flaky-ci uses (the gh api .../actions/workflows/{file}/runs REST calls from its Step 1 — not gh run list --json, for the same version-fragility reason given there):

# Vitest: how often does this exact test appear as FAIL across recent runs
# of the job that hosts it?
gh api "repos/growilabs/growi/actions/workflows/ci-app.yml/runs?per_page=50&status=completed" \
  -q '.workflow_runs[] | {databaseId: .id, conclusion, headSha: .head_sha, createdAt: .created_at}'

# Playwright: how often does this spec/browser show up in Retry#/flaky
# evidence across recent run-playwright jobs?
gh api "repos/growilabs/growi/actions/workflows/ci-app-prod.yml/runs?per_page=50&status=completed" \
  -q '.workflow_runs[] | {databaseId: .id, conclusion, headSha: .head_sha, createdAt: .created_at}'

Those calls return run ids only, so list each candidate run's jobs first — the log endpoint below is addressed by {JOB_ID}, never by a run id:

gh api "repos/growilabs/growi/actions/runs/${RUN_ID}/jobs" \
  -q '.jobs[] | {id, name, conclusion}'

Pick the job whose name hosts this identity (the vitest job that runs the spec, or the run-playwright shard for this browser — its name carries the browser, e.g. test-prod-node24 / run-playwright (chromium, 1/2, 8.0)).

For each candidate run, fetch that job's log (same method Step 0 of flaky-ci-routine.md decided — gh api --allow-escape-sequences "repos/growilabs/growi/actions/jobs/{JOB_ID}/logs" piped through sed -E 's/\x1b\[[0-9;]*[A-Za-z]//g', or mcp__github__get_job_logs, whichever this run is using; detect-flaky-ci Step 2 gives both commands in full) and grep for the test title. Do not reach for gh run view --job {JOB_ID} --log-failed here: on a run that was re-run it exits 0 and hands back the latest attempt's log rather than the attempt the job id belongs to, so a failure investigated from it reads as a pass, with no error to notice — detect-flaky-ci Step 2 records the measured case. Build a picture of: how often it fails, whether the failure mode is identical every time (points to a deterministic race, easier to fix) or varies (points to genuine timing noise), and — critically — read the stack trace / error origin in each occurrence: an error surfacing from a service/model file (not the spec itself) is your first clue toward a product-code race (see Step 3).

Read the spec file and the code path it exercises. For Playwright specs, static-analyze for timing-dependent patterns (missing await expect(...).toBeVisible() before interaction, reliance on fixed waitForTimeout, selectors matching multiple elements — see the comments.spec.ts "strict mode violation: resolved to 2 elements" pattern, a real example already seen in this repo's CI).

Second tier — measure a failure rate with the same repro workflow (always available, no local services needed, and stronger than local reproduction). The confirmation gate above spends 3 runs on a yes/no question ("does this spec ever pass on unchanged code?"). Root-cause work sometimes needs a rate instead: "failed 1/10" and "failed 9/10" point at very different mechanisms — the first looks like genuine timing noise, the second like a deterministic bug that merely presents as CI-only — and they land on different rows of Step 4's confidence table.

To take that measurement, push a second request exactly as in 2-B…2-F, on its own branch (flaky-repro/issue-{ISSUE_NUMBER}-rate), with a higher Flaky-Repro-Repeat — 10 is the workflow's cap. It runs on GitHub Actions' own MongoDB replica set and Elasticsearch rather than on whatever local environment this skill happens to be running in.

This is a different budget from the confirmation gate, and it costs real CI minutes: take it once, and only when the rate would actually change the Step 3 category or the Step 4 confidence. A flaky/confirmed issue that arrives without a confirmation measurement (see Precondition) can use this tier directly when a rate is what the root-cause read is missing.

Bonus tier — live reproduction, only if the current environment supports it. Probe rather than assume:

# vitest / integ tests need a real MongoDB replica set
mongosh --eval "db.adminCommand('ping')" 2>/dev/null || echo "no mongo"
# playwright needs installed browsers
pnpm exec playwright --version 2>/dev/null && ls ~/.cache/ms-playwright 2>/dev/null

If available:

# vitest
cd apps/app && pnpm vitest run {SPEC_PATH_PARTIAL} --repeat=20

# playwright
cd apps/app && pnpm playwright test {SPEC_PATH_PARTIAL} --repeat-each=10

A local reproduction (or local pass after a fix) is strictly additional confirmation on top of the CI-log evidence — never required to reach HIGH confidence, and its absence is never grounds to lower confidence below what the accumulated CI evidence alone supports.

Step 3: Root-Cause and Classify

Determine which category the flake belongs to — this decides the fix strategy in Step 4. Do not default to "just flaky, add a retry" — a race condition in product code surfaced by a test is a real bug, not a test problem, even though it manifests as flakiness. The same discipline applies to "Environment timing": do not default to "just bump the timeout" either — see the guardrail below.

Use git log --oneline -20 -- {SPEC_PATH} and read the code under test to tell "shared state" from "product race" — a test that fails only when run after a specific sibling usually points at a fixture; a test that fails with an error surfaced from a service/model file (not the spec file itself) in the log's stack trace usually points at a product-code race worth reading carefully before dismissing as test flakiness.

When the cause turns out to be deterministic: close the issue, don't fix a flake

None of the four categories fits a failure that is not actually non-deterministic. If the root-cause work lands on a cause that fails every time the conditions hold, the tracking issue is tracking the wrong thing: it is a real defect that happened to be found by this pipeline, and the flaky-tracking record should end rather than acquire a "fix". The shapes this repo has actually seen:

The tell is that it reproduces on demand. A repro tally of - Failed: 3 / - Runs: 3 on unchanged master (2-E's "possible genuine regression" reading) is the usual way this arrives, but a flaky/confirmed issue can land here too once the mechanism is understood.

Close it out — a conclusion, not a pause, so no flaky/needs-decision:

gh api "repos/growilabs/growi/issues/${ISSUE_NUMBER}/comments" -X POST -f body="$(cat <<'EOF'
### Closed: deterministic cause, not flaky

{what fails, and the mechanism that makes it fail every time}

{the evidence: the repro tally with its `- Runs:` / `- Failed:` values and
the workflow run URL, the lockfile/artifact/commit that carries the cause,
and the failing run(s) this issue already records}

{DECISION_RECORD — only on a human-decision resume: one paragraph naming the
option the human chose and linking their comment URL. Omit this line
entirely when this skill reached the close on its own evidence.}

This is not non-deterministic, so it is not tracked as a flaky test any
further. The underlying cause is handled as an ordinary issue / PR.

_Generated by [Claude Code](https://claude.ai/code)_
EOF
)"

The ### Closed: deterministic cause, not flaky heading is fixed — spell it exactly, in the comment's first line. It is one of the headings deliberately excluded from the occurrence match (flaky-ci-routine.md's dashboard step defines which headings count; that definition is the only mechanism), which is why this comment starting with ### does not contradict 2-E's bold-lead-in rule: 2-E is about not colliding with a counted heading.

{DECISION_RECORD} is the one part of the template that is present on some closes and absent on others: fill it in when a human decision sent this run down the close path (Resuming after a human decision, item 3 — this comment is then the first one the resumed path posts, so it is where that record has to live), and leave the line out entirely when the evidence alone led here. Everything else in the template is always written.

Never write "Fixed by" in this comment — that phrase is detect-flaky-ci's marker for "a fix for this tracking issue landed", and this close is not that: nothing was fixed here. You may link the separate issue / PR that handles the underlying cause, and doing so is useful: detect-flaky-ci's CLOSED-issue path, when it finds any PR reference, uses that PR's merged_at as the resolution instant, which is exactly the right line to draw — a later recurrence from before that merge is pre-fix evidence, one from after it is a genuine regression. With no PR reference at all the resolution time is simply unknown, and detect errs toward reopening.

Then swap the phase label and close as not planned:

LABELS_FILE="${TMPDIR:-/tmp}/flaky-deterministic-labels-${ISSUE_NUMBER}.json"

gh api "repos/growilabs/growi/issues/${ISSUE_NUMBER}" -q '[.labels[].name]' \
  | jq -c --arg rm '{EXACT_PHASE_UNDER_INVESTIGATION_LABEL}' \
          --arg add '{EXACT_PHASE_WONTFIX_LABEL}' \
          '{labels: ((map(select(. != $rm)) + [$add]) | unique)}' > "$LABELS_FILE"

gh api "repos/growilabs/growi/issues/${ISSUE_NUMBER}" -X PATCH --input "$LABELS_FILE"

gh api "repos/growilabs/growi/issues/${ISSUE_NUMBER}" -X PATCH \
  -f state=closed -f state_reason=not_planned

Fetch the exact label names first, as the Precondition requires — {EXACT_PHASE_WONTFIX_LABEL} reads ⏏ phase/wontfix today, and like every phase label it begins with an emoji, which is also why the whole-array PATCH is used here rather than a DELETE /labels/{name} (6-C explains that choice in full; follow it, do not re-derive it). The wontfix phase is the right one and {EXACT_PHASE_RESOLVED_LABEL} is not: nothing was resolved by this investigation. The tier label (flaky/confirmed / flaky/suspected) stays — it is the record of what this issue was opened as.

state_reason=not_planned is what marks the closure as "not going to be handled here", as opposed to a fix having landed. If the deterministic cause is worth fixing, that fix is an ordinary issue or PR: open it separately and link it from the close comment's evidence, never as "Fixed by".

Steps 4 through 6 do not run after this close. There is no fix branch, no repro measurement of a fix, and no PR from this skill.

If the identity fails again later, detect-flaky-ci's CLOSED-issue path reopens this issue, escalates it straight to flaky/confirmed and puts {EXACT_PHASE_NEW_LABEL} back on it — so it arrives at this skill looking like a fresh, already-confirmed flake. That is acceptable, and the close comment is what keeps it from being read that way: on picking up a reopened issue, read its comments for ### Closed: deterministic cause, not flaky before anything else. When it is there, the recurrence means "the deterministic cause is back" — the fix never merged, or it regressed — not "a flake needs confirming". Say so in Step 2 rather than spending a confirmation measurement to rediscover a cause this issue already names, and go to Step 3 with the recorded mechanism as the starting hypothesis.

Guardrail — a bare timeout increase is a stopgap; find what's actually driving the cost first

A numeric timeout bump (}, 15_000) → }, 20_000) and so on) is the cheapest possible edit, and that is exactly the problem: it is easy to reach for as the fix without ever asking what made the operation slow. Left unexamined, it tends to become the default "Environment timing" fix — and because it never touches the actual driver of the cost, it is usually a way of avoiding the root-cause work rather than doing it. It doesn't remove the underlying slowness; it just moves the load level at which the test starts failing again a bit higher, and the next CI-busy period trips it once more. Treat a bare bump as a last resort, not a first move — before proposing one, identify what is actually driving the observed duration. Two drivers show up repeatedly in this codebase; neither is fixed by a bigger number:

Driver 1 — the test's own wall time scales with a parameter of the test (a loop count, a data size, an iteration count) rather than being a fixed cost that merely got unlucky under load. Read the test body, not just the failing assertion. If it issues N real round-trips (network, DB, filesystem) where N is maxRequests, a loop bound, or similar, and the assertions of interest only concern the boundary (a limit being reached/exceeded, a count saturating, a last-element condition), that is a redesign opportunity, not an environment-timing dead end:

• Seed the precondition directly instead of looping to it — many libraries expose a lower-level primitive that reaches the same state in one call (e.g. rate-limiter-flexible's penalty(key, n) reaches n consumed points through the identical internal upsert path consume() uses, in a single round-trip, instead of looping consume() n times). The general shape: find the state-setting primitive the library already ships, confirm it goes through the same code path as the operation under test, and use it to jump straight to one step before the boundary.
• Exercise only the transition itself (the call(s) at and past the boundary) through the real code under test.
• This turns an O(N) cost into an O(1) cost, which removes the test's sensitivity to CI load rather than buying temporary headroom against it.

Driver 2 — the cost scales with how much concurrent setup work is in flight, not a test-local parameter. A beforeAll/afterAll in a setupFiles entry (migration replay, a singleton service cold-init, etc.) looks like it should run once per Vitest worker — the code usually memoizes a module-level singleton to make it look that way — but verify that before trusting it: Vitest's forks pool resets the module registry per test file under the default isolate: true, so a "per-worker singleton" comment can be describing intent, not actual behavior, and the cost can really be recurring on nearly every file all run long (confirm with a throwaway console.log inside the memoized branch and count how many times it fires across a few files — do not assume from the comment). Either way — once per worker or once per file — "this hook's own cost doesn't loop over anything, so there's nothing to redesign" sounds like it forces driver 1's dead end, but it doesn't: the cost still scales with how many files/workers pay it at the same moment, since anything else running on the same CI runner competes for the same CPU while each pays its own heavy setup cost (spawning a migration subprocess, initializing a singleton, etc.). A same-commit failure across several unrelated spec files that all timed out on the identical beforeAll line, with no code change near that line, is the signature of this shape (test/setup/crowi.ts's getInstance() and test/setup/migrate-mongo.ts in GROWI's own apps/app are the worked example — two successive fixes there each asserted the per-worker premise without checking it, and a console.log count across a few files falsified it). Where this applies:

• Measure the unloaded baseline, and don't stop at the number — check the model it's measuring. Run the failing spec(s) alone (no sibling workers contending) and read the hook's own duration off the reporter. If it already sits close to the current limit, a larger timeout may be warranted — state the measured number. If it sits well under the limit (e.g. under a fifth of it), the limit was never the problem; contention pushed a normally-fast hook past it. But a low unloaded number only says the hook is fast once — it does not tell you whether it pays that cost once per worker or once per file. Confirm which, per the paragraph above, before describing the mechanism in a fix or a doc: "once per worker" and "once per file, many times over" call for the same lever (bound concurrency) but very different claims about how much it helps, and an unverified "once per worker" claim is exactly the kind of plausible-but-unchecked assertion this whole guardrail exists to catch.
• Check for a same-project precedent that a bump already failed. If a different hook sharing the same setupFiles/project already had its timeout raised for the same "environment timing" reason and still flaked afterward (grep prior flaky-test issues/PRs for the project name), that is direct evidence the bottleneck is concurrency, not any single hook's deadline — proposing another bump for a second hook in the same project repeats a fix this repo's own history already falsified.
• Prefer bounding worker concurrency over widening the deadline. When the mechanism is "N workers/files doing cold-init at once fight for the runner's cores," the fix that addresses the mechanism is capping concurrent workers (Vitest poolOptions.forks.maxForks / poolOptions.threads.maxThreads, sized off availableParallelism() with headroom for sibling CI services like a DB/search container) for that project — not a bigger number on the hook that happened to lose the race this time.
• Never scope the change wider than the evidence. hookTimeout (and testTimeout) are configurable per-project in vitest.workspace.mts and per-hook as a call's last argument. A fix aimed at one shared setup hook must not raise the timeout for the whole project — that silently triples (or more) the failure-detection window for every other beforeAll/afterAll in every spec file the project covers, including a future genuine hang unrelated to this flake.

Both drivers reduce to the same discipline: find what the time is actually being spent on, and address that — a scaling parameter to redesign around, or contention to bound — rather than reaching for the timeout value because it's the line the stack trace happened to point at.

Only accept a bare timeout bump when:

• neither driver applies and no other driver can be identified after actually reading the test/hook and its call path (rare — most cases reduce to one of the two above), or
• as a modest safety margin layered on top of a fix that already addressed the real driver — not as the fix itself. State explicitly why the margin is still there (e.g. "no CI history yet for this reshaped test, and local/CI timing can differ") rather than leaving it unexplained.

A proposed fix that is only a numeric timeout change, with no evidence either driver was checked, is not HIGH confidence at Step 4 regardless of how clean the diff looks — see the confidence table there. Route it through the MEDIUM path and present the redesign or concurrency-capping alternative explicitly rather than defaulting to the bump.

Step 4: Fix-Approach Decision Gate

Confidence assessment:

In autonomous mode:

• HIGH → proceed to Step 5 automatically. State the category and planned fix.
• MEDIUM or LOW → pause per Pausing for a human decision. The comment presents the reproduction result, the suspected category and why confidence isn't HIGH; the - Recommendation: line names one of: 1) proceed with the best-guess fix, 2) quarantine with a comment linking this issue (only ever a stop-gate outcome, never an autonomous default), 3) escalate for human review of the product code, 4) close as unable to reproduce after N cycles (only if reproduction attempts genuinely found nothing across every method in Step 2), 5) close as a deterministic cause (Step 3's close path), when the evidence points that way but the mechanism is not certain enough to close it outright. Give the other options as prose above that line — the line itself carries the recommendation, not the menu.

In interactive mode: always ask.

When a human decision has already been handed to this run (see Resuming after a human decision), that decision crosses this gate: apply the option it chose and record it, instead of reassessing confidence and pausing again at the same place.

Quarantine guardrail: never mark a test .skip/.todo as the autonomous-HIGH default outcome. Quarantine is legitimate only as an explicitly chosen MEDIUM/LOW-gate outcome (interactive approval, or autonomous only when the category is "Environment timing" with no code path to fix), and the quarantine commit must reference this issue number in a comment so it is discoverable later.

Step 5: Implement

5-A: Branch

ISSUE_NUMBER=11823
FIX_BRANCH="fix/flaky-${ISSUE_NUMBER}-{short-description}"   # short slug, [a-z0-9-]

git fetch origin master
git checkout -b "$FIX_BRANCH" origin/master

Cut it from origin/master for the reason 2-B spells out: a workflow only runs if its file exists on the pushed ref, so a fix branch based on a commit that predates .github/workflows/flaky-repro.yml produces no flaky-repro check-run at all, and 6-B would then have nothing to gate on.

5-B: Fix

Apply the fix matching the Step 3 category. Whatever the category, the verification bar is higher than a normal bug fix: a flaky test that "looks fixed" after one green run has not been shown to be fixed.

Both essential-test-design and essential-test-patterns skills apply here — consult them if the fix touches test code, same as any other test change in this repo (.claude/rules/testing.md).

Verification comes before the PR, not after it. The fix commit carries a Repro Request of its own (5-C), so pushing the branch measures the fix the same way Step 2 measured the flake: the repro workflow replays the spec three times against the fixed code and posts the tally to the tracking issue, while ci-app.yml runs the ordinary suites on the same commit. Step 6 reads both and only then decides whether a PR is opened at all. If the environment happens to support local execution (probed in Step 2), run that too as a fast pre-push sanity check, but it never substitutes for the measurement in Step 6.

5-C: Commit

The fix commit's message ends with the Repro Request trailer block — that is what makes the push measurable. Git reads only the last paragraph of a message as trailers, so Fixes #{ISSUE_NUMBER} may come before the block but never after it, and the five trailers must be consecutive lines inside a single -m: splitting them across several -m flags puts each in its own paragraph and the workflow then sees only the last one (the one-paragraph rule is the Repro Request / Result contract in .kiro/specs/ci-flaky-test-detection's design.md).

The same applies to this session's own signature lines. Commits made from Claude Code end with Co-Authored-By: and Claude-Session: trailers, and those must go inside that same final paragraph, alongside the Flaky-Repro-* lines — never as a paragraph of their own after them. A blank line between the two groups makes the signature the last paragraph, which means git reports only Co-Authored-By / Claude-Session as trailers, the workflow sees no Flaky-Repro-Spec, and the push measures nothing. Use the exact signature lines this session was given, in place of the placeholders below:

# $ISSUE_NUMBER is the one 5-A set.
SPEC_PATH='src/server/routes/apiv3/g2g-transfer-replace-procedure.exclusive.integ.ts'
PROJECT='app-integration-exclusive'   # derived with 2-A's table

git commit -m "$(printf 'fix(scope): stabilize flaky test — %s\n\nFixes #%s\n\nFlaky-Repro-Spec: %s\nFlaky-Repro-Project: %s\nFlaky-Repro-Mode: file\nFlaky-Repro-Repeat: 3\nFlaky-Repro-Issue: %s\nCo-Authored-By: %s\nClaude-Session: %s\n' \
  '{short root cause}' "$ISSUE_NUMBER" "$SPEC_PATH" "$PROJECT" "$ISSUE_NUMBER" \
  '{Co-Authored-By value for this session}' '{Claude-Session URL for this session}')"

# Confirm git sees all of them as one trailer block before pushing:
git log -1 --format='%(trailers:only=true)'

That last command is the check that matters: it must list the five Flaky-Repro-* lines. If it prints only the signature lines, the paragraph split happened — amend the message rather than pushing.

Flaky-Repro-Spec and Flaky-Repro-Project are derived exactly as in 2-A (path relative to apps/app, project from the filename suffix), and Mode: file / Repeat: 3 are the same defaults — here they ask a different question: not "does this spec ever pass" but "does it pass three times out of three on the fixed code".

2-A's two exceptions apply here too, and they matter more at this step because an issue that arrives already flaky/confirmed reaches Step 5 without ever passing through 2-A. Apply the table and both exceptions: a path matching none of the four suffixes (a shared setup-hook identity like test/setup/migrate-mongo.ts) and a spec under src/features/growi-vault/__tests__/ both have no usable project. When no project can be derived, do not push the fix branch — an unmeasurable push cannot clear 6-B's gate, so there is nothing for it to lead to. Record "confirmation not measured (no repro project for this path)" and pause per Pausing for a human decision instead, leaving the fix uncommitted on the working tree or held on an unpushed branch for the human to decide on.

A Playwright fix carries no trailers. The repro workflow's project allowlist is vitest-only, so there is nothing for it to replay. Commit such a fix with the plain message (fix(scope): … plus Fixes #{ISSUE_NUMBER}) and take the Playwright row of 6-B; a trailer-less push to fix/flaky-** is treated as "no request" and its job exits 0 without measuring anything — a deliberate choice in the workflow (see the request step's comments in .github/workflows/flaky-repro.yml) so that a Playwright PR is not painted red by a workflow with no work to do.

Step 6: Verify via Real CI, Then the PR Readiness Gate

The fix branch already carries its own Repro Request (5-C), so verification happens on the branch, before any PR exists: 6-A waits for both measurements on the fix commit, 6-B decides from them whether a PR is opened at all, and 6-C opens it — ready for review from the moment it is created.

Step 5 and 6-A through 6-C share shell variables ($ISSUE_NUMBER, $FIX_BRANCH, $FIX_SHA, $CHECKS_FILE, $GATE_FACTS_JSON, $runs, $failed). Run them as one script; in a fresh shell, re-establish ISSUE_NUMBER and FIX_BRANCH from the values 5-A used and FIX_SHA with git rev-parse, rather than inventing new ones. The one pair that must never be split is the PR creation and its **Fix PR**: marker comment in 6-C.

6-A: Push the fix and wait for both measurements

# $ISSUE_NUMBER and $FIX_BRANCH are the ones 5-A set.
IS_PLAYWRIGHT_FIX=false   # true only for a Playwright identity, which asks for no measurement (5-C)

git push -u origin "$FIX_BRANCH"
FIX_SHA=$(git rev-parse HEAD)

CHECKS_FILE="${TMPDIR:-/tmp}/flaky-fix-checks-${ISSUE_NUMBER}.json"
STARTED=$(date +%s)
STARTUP_GRACE=$(( STARTED + 5 * 60 ))
DEADLINE=$(( STARTED + 45 * 60 ))

while :; do
  node bin/flaky-ci/scripts/check-runs-facts.ts --sha "$FIX_SHA" > "$CHECKS_FILE"

  repro_status=$(jq -r '.flakyRepro.status' "$CHECKS_FILE")
  ci_total=$(jq -r '.ciApp.total' "$CHECKS_FILE")
  # Still-pending ci-app-* checks: check-runs-facts.ts already deduped to the
  # newest per name (a commit carries two entries per job once a PR exists —
  # a push event and a pull_request event — and a superseded push leaves a
  # `cancelled` one behind), so `.checks[]` here can never double-count a
  # stale entry the way reading every raw check-run would.
  ci_pending=$(jq '[ .checks[] | select(.name | startswith("ci-app-")) | select(.status != "completed") ] | length' "$CHECKS_FILE")
  now=$(date +%s)

  if [ "$ci_total" -gt 0 ] && [ "$ci_pending" -eq 0 ] \
     && { [ "$repro_status" = 'completed' ] || [ "$IS_PLAYWRIGHT_FIX" = true ]; }; then
    break
  fi

  # Nothing started within the grace period: the trigger never matched, and
  # waiting out the other 40 minutes would only delay the same answer.
  if [ "$now" -ge "$STARTUP_GRACE" ]; then
    if [ "$ci_total" -eq 0 ]; then
      echo 'no ci-app-* check-run within 5 minutes — not measured'
      break
    fi
    if [ "$IS_PLAYWRIGHT_FIX" != true ] && [ "$repro_status" = 'absent' ]; then
      echo 'no flaky-repro check-run within 5 minutes — not measured'
      break
    fi
  fi

  if [ "$now" -ge "$DEADLINE" ]; then
    echo 'wait capped at 45 minutes'
    break
  fi
  sleep 60
done

jq -r '.checks[] | select(.name == "flaky-repro" or (.name | startswith("ci-app-")))
       | "\(.name)\t\(.status)\t\(.conclusion)"' "$CHECKS_FILE"

Which checks count. Take every check-run whose name starts with ci-app-; do not hardcode a list. Today ci-app.yml produces ci-app-lint (24.x), ci-app-test (24.x, 6.0) and (24.x, 8.0), ci-app-test-integration (24.x, 8.0, 8, 8.19.16) and (24.x, 8.0, 9, 9.3.3), and ci-app-launch-dev (24.x, 6.0) and (24.x, 8.0) — but those matrix cells change with the supported Node and MongoDB versions, and the ci-app- prefix is the stable part. test-prod-node24 / … (ci-app-prod.yml) is not part of this gate: that workflow's push trigger is an allowlist of master and dev/*, so it never runs on a fix branch.

45 minutes is the cap for the whole wait, not per check: ci-app-test-integration alone usually takes 20–30 minutes, while the repro workflow's three file-mode runs finish long before it.

Two endings leave the wait without a measurement. Both route to 6-B's "not measured" row, and both are visible within about five minutes — which is why the loop leaves after $STARTUP_GRACE instead of burning the full 45:

• no flaky-repro check-run appears — the branch name did not match fix/flaky-**, or its base predates flaky-repro.yml (5-A);
• no ci-app-* check-run appears — ci-app.yml's push trigger has a paths filter, so a fix touching nothing under apps/app/**, packages/** or the listed root files starts no run at all.

6-B reads the repro tally itself (pr-gate-facts.ts, below) — nothing more is fetched here.

6-B: The PR gate — decided here, and only here

A PR is created if and only if all three of these hold for $FIX_SHA:

1. the ### Repro result comment pinned to $FIX_SHA exists and reads - Failed: 0, with - Runs: at least the repeat that was requested (3 by default);
2. at least one ci-app-* check-run exists on $FIX_SHA, and every one of them has conclusion == "success" (an empty set is "nothing ran", not "nothing failed" — it never passes this condition);
3. the diff touches only what Step 3 identified: a test-side fix stays inside the spec file and its fixtures; a product-side fix stays inside the module Step 3 named. Anything wider is MEDIUM — the tally proves the spec is stable, not that the extra code is right.

Anything else means no PR. There is no partial credit and no second place where this call is made: 6-C assumes the gate has already been passed.

git fetch origin master
GATE_FACTS_JSON="${TMPDIR:-/tmp}/flaky-fix-gate-${ISSUE_NUMBER}.json"

node bin/flaky-ci/scripts/pr-gate-facts.ts \
  --issue "$ISSUE_NUMBER" --sha "$FIX_SHA" --base origin/master > "$GATE_FACTS_JSON"
GATE_FACTS_STATUS=$?

runs=$(jq -r '.tally.runs // "none"' "$GATE_FACTS_JSON")
failed=$(jq -r '.tally.failed // "none"' "$GATE_FACTS_JSON")
ci_not_success=$(jq -r '[ .ciApp.notSuccess[] | "\(.name)=\(.conclusion)" ] | join(", ")' "$GATE_FACTS_JSON")
scope=$(jq -r '.changedFiles[]' "$GATE_FACTS_JSON")

echo "repro:  Runs=${runs}  Failed=${failed}"
echo "ci-app not success: ${ci_not_success:-none}"
echo "files changed by the fix:"; echo "$scope"

pr-gate-facts.ts fetches the repro comment and the check-runs itself (it does not read $CHECKS_FILE from 6-A's wait loop) — see bin/flaky-ci/README.md's contract row for exactly which reading it produces for each case. Two readings its output can carry that are not failures of this call: tally is null (not the same as $GATE_FACTS_STATUS being non-zero) when the issue simply has no ### Repro result comment pinned to $FIX_SHA yet — condition 1's own "not measured" row below reads that — and ciApp.total: 0 when no ci-app-* check-run exists at all, which condition 2 treats as failing, not as an error. $GATE_FACTS_STATUS itself being non-zero (exit 2) means the call could not read anything at all — the comments fetch, the check-runs fetch, or the local git diff against --base failed outright — and none of conditions 1–3 can be read from $GATE_FACTS_JSON in that case.

Condition 3 is read from $scope (changedFiles[]), computed as a three-dot diff against origin/master inside the script: what the fix commits changed since they left master, not also every file master gained after the branch point (which would fail the condition on any fix branch that is merely behind master).

Why the check-run's own conclusion cannot stand in for condition 1. A push to fix/flaky-** whose commit carries no trailers is treated as "no request": the job exits 0 and its check-run is success having measured nothing. A comment that never arrived (a locked issue, a momentary GitHub outage) leaves the same green check-run behind, since the posting step deliberately does not fail the job. So success here means "nothing broke", never "the fix was measured" — the - Failed: and - Runs: lines are the evidence, and their absence is a "not measured" reading.

A Playwright fix takes this gate with condition 1 removed (that is what IS_PLAYWRIGHT_FIX=true marks in 6-A), because it requested no measurement: conditions 2 and 3 open its PR, and the Verification section says where the real evidence comes from instead (6-C). The flaky-repro check-run on such a commit is the no-op success just described — ignore it.

Autonomous: HIGH → 6-C. MEDIUM or LOW → no PR; pause as below. Interactive: always ask, using the same readings. A human decision handed to this run (see Resuming after a human decision) crosses this gate the same way it crosses Step 4's.

When the gate is not passed, leave the branch pushed — it is the evidence — and pause per Pausing for a human decision. This is the body to give that pause comment; the block itself owns the two closing lines, the flaky/needs-decision label and the "stop, watch nothing" part:

**Fix gate not passed** — no PR was opened.

- Fix commit: `${FIX_SHA}` on `${FIX_BRANCH}`
- Condition 1 (repro tally): {`- Runs:` / `- Failed:` for this commit, or "no result comment for this commit" when `tally` is `null`} ({check-run URL})
- Condition 2 (normal CI): ${ci_not_success:-all ci-app-* success}
- Condition 3 (scope): {the files the diff touches, against what Step 3 identified}
- {one-line reading of which condition failed, pointing at the excerpt in the result comment when there is one}

$FIX_SHA/$FIX_BRANCH are 6-A's variables and $ci_not_success is this step's own (read from $GATE_FACTS_JSON, not $CHECKS_FILE), so post it from the same shell they live in, with the same gh api "…/comments" -X POST -f body="$(cat <<EOF … EOF)" form 2-E uses — an unquoted EOF so those variables expand, which means the backticks above have to be escaped as \` inside it.

6-C: Open the PR, ready for review, by REST

The gate is 6-B; do not re-derive the condition here.

Create the PR and post the marker comment in the same shell invocation — $PR_HTML_URL lives only as long as the shell that set it, and a separate tool call starts a fresh one with no memory of it:

PR_BODY_FILE="${TMPDIR:-/tmp}/flaky-fix-pr-body-${ISSUE_NUMBER}.md"
REPRO_RUN_URL=$(jq -r '.tally.workflowRunUrl' "$GATE_FACTS_JSON")
CI_RUN_URL=$(gh api "repos/growilabs/growi/actions/workflows/ci-app.yml/runs?head_sha=${FIX_SHA}&per_page=1" \
  -q '.workflow_runs[0].html_url')

cat > "$PR_BODY_FILE" <<EOF
## Summary

{description of the fix}

## Root Cause

{category from Step 3 + specific mechanism}

## Verification

- Repro workflow on this fix commit (\`${FIX_SHA}\`): \`- Runs: ${runs}\`,
  \`- Failed: ${failed}\` — the spec was replayed ${runs} times against the
  fixed code without a single failure. ${REPRO_RUN_URL}
- Normal CI on the same commit: every \`ci-app-*\` check \`success\`. ${CI_RUN_URL}

Fixes #${ISSUE_NUMBER}
EOF

PR_HTML_URL=$(gh api repos/growilabs/growi/pulls -X POST \
  -f title="fix: stabilize flaky test in {short scope}" \
  -f head="$FIX_BRANCH" \
  -f base=master \
  -F draft=false \
  -F "body=@${PR_BODY_FILE}" \
  -q '.html_url')

gh api "repos/growilabs/growi/issues/${ISSUE_NUMBER}/comments" -X POST \
  -f body="**Fix PR**: ${PR_HTML_URL}"

-F draft=false is the whole of the readiness story: -F converts the literal false into a JSON boolean, which is what the endpoint's draft field expects — use -F here, not -f, which sends every value as a string. -F "body=@<path>" reads the field's value from a file, which keeps a multi-paragraph Markdown body out of the argument list.

The marker must be its own comment — a fixed one-line marker, never appended to another comment — and its exact text must be **Fix PR**: {PR_HTML_URL}. That exact string is what the Dashboard Updater matches on, so do not add a heading or extra wording that would break the match, and do not reuse detect-flaky-ci's ### Additional observation / ### Backfilled observation headings here: this comment is deliberately excluded from that issue's observation count.

For a Playwright fix, replace the Verification section with this — there is no repro tally to quote, and the end-to-end evidence arrives later:

## Verification

- Normal CI on this commit: every `ci-app-*` check `success`. {ci-app run URL}
- End-to-end verification is delegated to the `run-playwright` job in this
  PR's CI pipeline. It does not run on a fix-branch push, and
  `reusable-app-prod.yml` gates it on `head_ref` starting with
  `mergify/merge-queue/`, so it executes when this PR enters the merge queue,
  with Playwright's own `retries: 2` on top. If the spec still needs a retry
  there, `detect-flaky-ci` observes it and the tracking issue comes back.

Then move the issue's phase label. Read the current set and PATCH it back as one array:

LABELS_FILE="${TMPDIR:-/tmp}/flaky-fix-labels-${ISSUE_NUMBER}.json"

gh api "repos/growilabs/growi/issues/${ISSUE_NUMBER}" -q '[.labels[].name]' \
  | jq -c --arg rm '{EXACT_PHASE_UNDER_INVESTIGATION_LABEL}' \
          --arg add '{EXACT_PHASE_RESOLVED_LABEL}' \
          '{labels: ((map(select(. != $rm)) + [$add]) | unique)}' > "$LABELS_FILE"

gh api "repos/growilabs/growi/issues/${ISSUE_NUMBER}" -X PATCH --input "$LABELS_FILE"

Fetch the exact label names first (gh api repos/growilabs/growi/labels --paginate -q '.[].name', as the Precondition already requires) — never hardcode them. The whole-array PATCH is used here instead of 2-E's POST /labels plus DELETE /labels/{name} pair because the phase labels begin with an emoji (1️⃣, 5️⃣), which a DELETE would have to carry percent-encoded in the URL path, while PATCH keeps the name in the JSON body. PATCH replaces the entire set, so it must be computed from the labels read immediately before it — which is also what keeps flaky/confirmed on the issue.

6-D: There is no ready-flip step

The PR is ready for review the moment 6-C creates it, so nothing follows. Turning a draft PR into a ready one is markPullRequestReadyForReview, a GraphQL-only mutation with no REST equivalent, and a cloud routine's session blocks GraphQL. Creating the PR non-draft removes that dependency entirely: readiness is decided by 6-B's gate, before the PR exists, rather than by a state change afterwards.

flaky/confirmed stays on the issue — it is a permanent record that this was a real, confirmed flake, not something to remove on resolution. If the same identity key resurfaces after this merges, detect-flaky-ci's "existing CLOSED issue found" path reopens it automatically; that recurrence check is the long-term backstop this skill's verification ultimately relies on, on top of 6-B's tally.

Routine discipline — no subscriptions, no notifications, no wake-ups

This skill must not subscribe to PR or issue events, must not request a mobile notification, and must not schedule a wake-up — no timer, no reminder, no "check back in N minutes", no background session left watching a PR, an issue or a workflow run. That holds at every exit: an issue paused for a human decision, a PR just opened in 6-C, a flaky-repro run still going when the 30- or 45-minute cap expired. None of them is watched afterwards. flaky-ci-routine.md states the same rule for the whole routine and gives the failure it exists to prevent; this skill is bound by it whether it was invoked from that routine or on its own.

The next routine run is the entire recovery mechanism. It re-selects a paused issue once a human has answered, closes what has gone quiet, and shows the rest on the dashboard. Nothing here is urgent to the minute.

The one wait this skill does perform — the bounded polling of check-runs in 2-C and 6-A (60-second interval, 30- and 45-minute caps) — is not a subscription: it blocks the step that started it, ends by itself, and leaves nothing behind. When it runs out, the outcome is "not measured" and the issue pauses. Never convert that timeout into a wake-up.

The signature this skill puts on its comments, and the flaky/needs-decision label, are defined in flaky-ci-routine.md's Shared constants — see Every comment this skill posts is signed for how they are used here.

Error Handling

• REST (gh api) is the form every read and write in this skill uses — not a fallback to reach for after a failure. gh issue / gh label / gh pr subcommands are GraphQL mutations, and a cloud routine's session rejects GraphQL ("This GraphQL query is not enabled for this session"), so seeing that error means a subcommand slipped into a step: replace that call with the gh api form the step documents and continue — do not retry the subcommand. detect-flaky-ci's Error Handling states the same rule for its own writes.
• Issue is neither flaky/confirmed nor flaky/suspected: stop, do not investigate (see Precondition). This is not a pause: nothing was investigated, so there is no decision for a human to make on the flaky pipeline's behalf — do not add flaky/needs-decision, and do not post a pause comment.
• The fix commit's flaky-repro check-run is absent or failure, or it completed without a ### Repro result comment carrying that commit's SHA (Step 6-A): the fix was not measured. Do not read a success conclusion as verification — a trailer-less push and a comment that failed to post both leave the check green — and do not let Step 2's confirmation tally or a fully green ci-app-* set stand in for it. Open no PR, and pause per Pausing for a human decision with what happened as the comment's body. (A Playwright fix is the one exception, and only because it requests no measurement: its gate is the ci-app-* set plus the scope condition — see Step 6-B.)
• A human approves proceeding with a best-guess fix at a MEDIUM gate for an issue that came in as flaky/suspected and whose confirmation measurement came back Failed == Runs (Step 2/Step 4): the label is still flaky/suspected at that point, not flaky/confirmed — do not silently promote it, and nothing later in this run changes that. In particular a - Failed: 0 tally at Step 6 is not grounds to promote it: that tally measures the fixed code, and a spec passing after a change says nothing about whether the original was non-deterministic. Leave the label as-is and let the PR body's Root Cause section carry the caveat that the flakiness was never empirically reproduced.
• Reproduction impossible in devcontainer (e.g. browser deps missing): fall back to log-based analysis, note the limitation, and do not let this alone push confidence below what the CI evidence already supports. This is not a pause either — it changes nothing about confidence, so it adds no label.
• Fix requires product-code changes with security/auth/data implications: treat as MEDIUM or LOW regardless of how clear the race looks — this skill is not a substitute for security-reviewer on sensitive code paths — which means Step 4 pauses per Pausing for a human decision, with escalation for human review of the product code as the recommended option.