Agent Skills
› mohitagw15856/pm-claude-skills
› agent-observability-spec
agent-observability-spec
GitHub为AI Agent或LLM应用生成生产级可观测性规范。定义Trace Schema、健康/成本/质量指标及告警阈值、采样保留策略和隐私说明，确保能追踪行为并快速定位故障。
触发场景
设计Agent的可观测性方案
确定LLM应用的日志记录内容
定义质量和成本监控指标
排查Agent异常行为
安装
npx skills add mohitagw15856/pm-claude-skills --skill agent-observability-spec -g -y
SKILL.md
Frontmatter
{
"name": "agent-observability-spec",
"description": "Specify the tracing, metrics, and alerting for an AI agent or LLM feature in production. Use when asked what to log for an LLM app, design agent tracing or spans, define quality and cost monitors, or answer 'how do we know if the agent is misbehaving?'. Produces an observability spec with a trace schema, metric definitions with owners and alert thresholds, sampling and retention policy, and a privacy note for logged content."
}
Agent Observability Spec Skill
You can't fix what you didn't record. For LLM systems the unit of observability is the trace — everything the model saw and did — because behaviour, not uptime, is what fails. This skill specifies what to capture, what to compute from it, and when to page someone.
What This Skill Produces
- A trace schema: per-request spans and the fields each must carry
- Metric definitions across health, quality, cost, and behaviour — each with a threshold and owner
- A sampling and retention policy that keeps cost sane and debugging possible
- A privacy note: what logged content contains, who can see it, and how long it lives
Required Inputs
Ask for (if not already provided):
- The system's shape — single LLM call, RAG pipeline, or multi-step tool-using agent
- Traffic volume and cost sensitivity — full tracing at 10M req/day is a budget decision
- What "misbehaving" means here — the two or three failure modes that matter most (wrong facts? wrong actions? cost? refusals?)
- Existing observability stack (Datadog, Langfuse, OTel, homegrown) — spec into it, not around it
Trace Schema
Every request produces one trace; every model call, retrieval, guardrail check, and tool execution is a span. Minimum fields:
| Span | Must capture |
|---|---|
| Request root | request id, user/session (pseudonymous), feature + prompt version, model id, total tokens, total cost, latency, terminal status |
| Model call | full input context (or content-addressed ref), output, finish reason, tokens in/out, cached-token share, temperature |
| Retrieval | query, top-k ids + scores, which chunks entered the context |
| Tool call | tool name, arguments, result (or ref), duration, error |
| Guardrail | check name, verdict, and what it did (blocked / rewrote / flagged) |
| User signal | edits, regenerates, thumbs, abandonment — joined to the trace id |
The test of the schema: an engineer can replay any incident from its trace alone (see agent-incident-postmortem).
Metrics and Alerts
Define four families; every metric gets a threshold, a window, and an owner.
- Health — error rate, p50/p95 latency, timeout rate, provider 429/5xx rate. Page on these.
- Cost — cost per request (p50, p99), tokens per request, cache hit rate, daily spend vs. budget (pair with
llm-cost-latency-budget). Alert on p99 and daily-budget burn — cost incidents are caused by the tail, not the mean. - Quality proxies — format/schema violation rate, refusal rate, groundedness-check failure rate, judge score on a sampled slice, regenerate/edit rate. Alert on drift vs. a rolling baseline: absolute thresholds go stale, deltas don't.
- Behaviour (agents) — steps per task, tool-error rate, loop detection (same tool + same args N times), unauthorised-action attempts caught by guardrails. Page on the last one.
Sampling & Retention
- Metadata for 100% of requests (ids, versions, tokens, cost, status) — this is cheap and non-negotiable.
- Full content traces: 100% for errors, guardrail hits, and negative user signals; [1-10]% random sample for the rest, adjusted to volume.
- Retention: full content [30-90] days, metadata [12+] months for trend baselines; incident traces pinned indefinitely.
- Privacy: logged context contains user data — state where it lives, who has access, how deletion requests reach it, and that traces are scrubbed or access-gated before wide sharing.
Output Format
Observability Spec: [feature/agent]
System shape: [calls/pipeline/agent] · Volume: [req/day] · Stack: [tooling]
Trace schema: [the span table, tailored]
Metrics:
| Metric | Family | Threshold / baseline | Window | Alert → owner |
|---|
Sampling & retention: [the policy]
Privacy: [content classification, access, deletion path]
Dashboards: [the 2-3 views: live health, quality drift, cost]
First incident drill: pick yesterday's worst trace and confirm it can be replayed end-to-end from the stored data.
Quality Checks
- Any incident is replayable from its trace alone — the schema was tested against that bar
- Every metric has a number, a window, and a named owner — no orphan dashboards
- Quality alerts are drift-based against a rolling baseline, not absolute guesses
- Sampling keeps 100% of error/guardrail/negative-signal traces
- The privacy note exists and names retention and access — logged prompts are user data
Anti-Patterns
- Do not log only inputs and outputs — without retrieval and tool spans, root cause analysis is guesswork
- Do not alert on mean cost or mean latency — the tail is where both incidents live
- Do not run judge-based quality scoring on 100% of traffic — sample; spend the budget on better baselines
- Do not treat observability as launch-week scaffolding — drift metrics only work with months of baseline
- Do not ship an agent that can take actions without logging the guardrail verdicts alongside the actions
版本历史
- a38bc30 当前 2026-07-05 11:10
