Agent Skills › anthropics/cwc-workshops › incident-triage-runbook

incident-triage-runbook

GitHub

SRE生产事故排查手册，用于调查延迟或错误率异常。指导按序检查部署、指标及日志，识别常见根因，并输出结构化结论与JSON报告以支持事后复盘。

ship-your-first-managed-agent/skills/incident-triage-runbook/SKILL.md anthropics/cwc-workshops

Trigger Scenarios

调查生产服务故障分析延迟尖峰或错误率升高询问特定服务异常原因

Install

npx skills add anthropics/cwc-workshops --skill incident-triage-runbook -g -y

More Options

Non-standard path

npx skills add https://github.com/anthropics/cwc-workshops/tree/main/ship-your-first-managed-agent/skills/incident-triage-runbook -g -y

Use without installing

npx skills use anthropics/cwc-workshops@incident-triage-runbook

指定 Agent (Claude Code)

npx skills add anthropics/cwc-workshops --skill incident-triage-runbook -a claude-code -g -y

安装 repo 全部 skill

npx skills add anthropics/cwc-workshops --all -g -y

预览 repo 内 skill

npx skills add anthropics/cwc-workshops --list

SKILL.md

Frontmatter

{
    "name": "incident-triage-runbook",
    "description": "The SRE team's runbook for triaging production latency and error-rate incidents. Use this whenever investigating an incident, a latency spike, elevated error rates, or when asked \"what caused X\" about a production service."
}

Incident triage

If you change the order below, say why in #sre.

Order of operations

Pull deploys for the last 6h. Don't open the log first.
Line the deploy timestamps up against p99_latency_ms / error_rate for the paged service. State the gap ("deploy 14:31, p99 moves 14:33").
If a deploy lines up: pull the diff, read it. Check for the stuff in the next section.
Then grep the log to confirm. Don't grep to fish.
No deploy lines up → the service may be a victim. Check db_pool_utilization across checkout/cart/auth/inventory, then the upstream deps (db-primary, db-inventory, idp.internal). Find what moved first.

Ask get_metrics for a name that doesn't exist and it lists the ones that do.

Things that have burned us

In rough order of how often:

per-row query where there used to be a batch
cache decorator removed "temporarily"
new query, no index
blocking call in an async handler
retry loop with no backoff

Write-up

One line at the bottom of your reply:

Root cause: <sha> — one sentence on the mechanism.

If it wasn't a deploy, put the component or upstream dep where the sha goes (db-primary, stripe-api, whatever). Still one sentence.

Everything above that line is evidence. Keep it short; the long version goes in the postmortem doc.

If you were handed an incident id, also save the structured version to /workspace/findings/<INCIDENT_ID>.json so the postmortem can be built from it:

{
  "incident_id": "INC-0000",
  "service": "checkout",
  "title": "one line, what users saw",
  "root_cause": {"ref": "<sha or component>", "summary": "one sentence on the mechanism"},
  "timeline": [{"time": "14:31:18", "event": "what happened"}],
  "evidence": ["one fact per line, with the number or timestamp that proves it"],
  "ruled_out": ["what you checked that turned out not to be the cause, and why"],
  "impact": {"metric": "p99_latency_ms", "baseline": 65, "peak": 3640, "unit": "ms"},
  "fix": "the immediate remediation",
  "caused_by_incident": null
}

Set caused_by_incident to another page's id when your service is a victim of it.