debugging-difficult-bugs — instrument, reproduce, read, then fix

Core idea: when you can't see the failure by reading code, make the runtime tell you. Add temporary append-only JSONL logging along the real code path, reproduce the real issue once, read the log chronologically, and only then fix. Never make a second speculative fix without new runtime evidence.

Announce at start: "Using the debugging-difficult-bugs skill — instrumenting the runtime path to observe the failure."

Step 1: State the uncertainty

Write down: what you believe, what you can't verify statically, and the exact runtime path that must be observed (route → service → query, posting, job).

Step 2: Add temporary unconditional instrumentation

Rules:

• Unconditional — never gated behind an env var, debug flag, or log level. If reproduction requires remembering to set a flag, it will silently not fire.
• Append-only JSONL, one JSON object per line, to a file in the process's working directory.
• Log boundaries and decisions, not every line: function entry/exit, branch decisions with the data that caused them, state before/after mutation, async ordering markers, caught errors, return-value shapes.
• Log shapes, not payloads: ids, keys, counts, statuses. Never log tokens, auth headers, cookies, or full user content.

import { appendFileSync } from "node:fs";
import { join } from "node:path";

function debugBug(event: string, data: Record<string, unknown> = {}) {
  appendFileSync(
    join(process.cwd(), "debug-difficult-bug.jsonl"),
    `${JSON.stringify({ ts: new Date().toISOString(), event, ...data })}\n`
  );
}

debugBug("service.beforeUpdate", { id, companyId, status: row.status });

Carbon multi-process note. The ERP and MES dev servers and the Inngest handlers run as separate processes with different working directories; postings (the ERP modules' service files) run in the ERP process, also when the MES calls them (over HTTP, erpPosting). Log process.cwd() + a process role once at startup, or use distinct filenames (debug-erp.jsonl, debug-jobs.jsonl). The Docker edge-runtime container only hosts the embedding edge function; there, console.error JSON lines (visible in container logs) stand in for a file.

Step 3: Reproduce the real issue once

• Prefer reproducing yourself: boot the stack (crbn up if not already running), authenticate with /auth, and drive the exact failing flow with agent-browser (the /test skill documents Carbon's form gotchas — requestSubmit, react-aria blur).
• If only the user can reproduce (their data, their environment), tell them exactly: "I added temporary logging. Reproduce the issue once, then point me at <cwd>/debug-difficult-bug.jsonl."

Step 4: Read the log BEFORE fixing

Read chronologically and answer, in writing:

1. Did the instrumented path actually run?
2. What was the expected sequence of events?
3. What was the actual sequence?
4. What is the first point where state/order/branch diverges from expectation?

That first divergence is the root cause candidate. Feed it back into the root-cause brief (or write one now) — then implement via /fix, whose failing regression test must assert the actual divergence you observed, not your earlier assumption.

Step 5: Clean up — mandatory

• Remove every temporary log call, helper, and import.
• Delete generated .jsonl files.
• Check the final diff explicitly for leftovers: git diff | grep -n "debugBug\|debug-difficult\|\.jsonl" → expect no hits.

The final diff contains only the fix and its tests.

Done when

• The first divergence point is identified from log evidence (quote the lines)
• The fix landed via /fix with a red→green regression test asserting that behavior
• Reproduction of the original flow now passes
• Zero instrumentation remnants in the diff