observability

Step 1: Map the failure modes

Before instrumenting, list what can go wrong:

1. What inputs could be invalid? External API responses, user-submitted data, filesystem state, env vars.
2. What external dependencies could fail? Network, DB, child processes, filesystem permissions.
3. What internal invariants could break? State transitions, lock acquisition, concurrency assumptions.
4. What silent corruption is possible? Truncated writes, partial transactions, stale caches.

This map tells you where to instrument. Don't instrument uniformly — instrument at the decision points where these failures would manifest.

Step 2: Structured logs at decision points

For each decision the code makes that could plausibly go wrong later:

• What decision? ("Dispatching unit X", "Retrying with backoff", "Skipping validation because flag set")
• Why? ("status=pending", "previous attempt exit=1", "--dev flag set")
• What would a future debugger want to know? (The values that drove the choice.)

Format:

log.info({
  event: "unit-dispatched",
  unitType: "slice",
  unitId: "S02",
  reason: "pending",
  attempt: 1,
  flowId,
});

Use the project's existing logger if one exists. In gsd-pi, follow the patterns in src/resources/extensions/gsd/activity-log.ts and src/resources/extensions/gsd/journal.ts — structured JSONL, one event per line, with ts, event, and domain-specific fields.

Avoid:

• console.log("here") — what does "here" mean in six months?
• Logging secrets, tokens, or PII — ever.
• Formatting structured data into a prose string — it can't be grepped or filtered.

Step 3: Persist failure state

When something fails in a way the caller can't immediately handle, write the failure state to disk:

await writeAtomically(
  resolve(".gsd/runtime/last-error.json"),
  JSON.stringify({
    ts: new Date().toISOString(),
    phase: "execute",
    unitId,
    error: { message, stack, code },
    retryCount,
  })
);

A fresh agent reading .gsd/runtime/ sees what happened last, what was retried, and where the process stopped. Pattern exists already in gsd-pi — reuse the atomic-write.ts helpers and the .gsd/runtime/ and .gsd/forensics/ directories.

Step 4: Health and status surfaces

For long-running processes:

• Health endpoint (HTTP server) or status file (CLI tool). Cheap to call, no side effects. Returns current state: {status: "healthy" | "degraded" | "down", ...diagnostics}.
• Digest view — a small representation of recent work. In gsd-pi, this is STATE.md and the health widget. In a server, it's /internal/status with last 10 request summaries.
• Minimal metrics — counters for the 3–5 things that matter (requests, errors, active jobs). Not everything — just what drives alerts.

Don't build a metrics empire. Build exactly what you'd check at 3am.

Step 5: Explicit failure modes

Replace silent handling with explicit:

// Bad
try {
  return await db.getUser(id);
} catch {
  return null;
}

// Good
try {
  return await db.getUser(id);
} catch (err) {
  log.error({ event: "db-getuser-failed", userId: id, err: serializeError(err) });
  throw new DatabaseError("Failed to load user", { cause: err, userId: id });
}

The caller now knows the failure happened, gets an error type it can branch on, and a log line exists for forensics.

Step 6: Remove the scaffolding

Before shipping, cull the ad-hoc instrumentation you used while debugging. Keep only:

• Decision-point logs that a future agent would use
• Persistent failure state
• Health/status surfaces
• Explicit failure modes

Drop:

• Temporary console.log debug lines
• Spammy per-iteration logs that no one will read
• Metrics that were "might be useful someday"

The system prompt says it plainly: "Remove noisy one-off instrumentation before finishing unless it provides durable diagnostic value."

Step 7: Verify the signals work

Pick one plausible failure mode from Step 1 and simulate it (inject an error, point at a missing file, break a dependency). Confirm:

1. The failure produced a log line a cold-start agent could understand.
2. The failure state persisted somewhere durable.
3. The health surface reflects the degraded state.
4. Nothing was swallowed silently.

If any signal is missing, add it — that's the gap this skill exists to catch.