pre-release-sweep
GitHub发布前全量回归测试技能,验证自上次标签合并至主分支的所有变更。包含CI门禁、环境搭建、Playwright浏览器遍历及缺陷修复,输出追踪文档。
触发场景
安装
npx skills add LanternOps/breeze --skill pre-release-sweep -g -y
SKILL.md
Frontmatter
{
"name": "pre-release-sweep",
"description": "Use before cutting a Breeze release, or whenever asked to \"test everything since the last release\", \"check what's changed on main in the browser\", \"boot the stack and verify the merged PRs\", or \"pre-release QA\". Covers the gate on in-flight CI, standing up a stack at current main, walking every merged-since-tag change through Playwright, tracking results in a dated doc, logging UI\/UX paper cuts, and fixing small defects on the way. For a whole-app sweep unrelated to a release window use `ui-qa-sweep`; for one just-built feature use `feature-testing`."
}
Pre-Release Sweep
Verify every change merged since the last release tag actually works in the
browser, on a stack built from current origin/main, and leave a tracking
doc a release cut can cite. The tracking doc is the deliverable; the browser
work exists to fill it.
Tracking doc: docs/testing/release-sweeps/<YYYY-MM-DD>-<from-tag>-to-<to>.md
(copy TEMPLATE.md in that directory). Write to it as you go — a sweep
outlives a context window.
Phase 0 — Gate on in-flight work
gh pr list --state open— non-draft PRs againstmainwith CI running are the "waiting" set. Draft/stacked PRs are not.gh pr checks <n> --watchin the background (never poll by hand).- Merge only on head-SHA green (bare
gh pr merge <N>enqueues; never--admin). Ignore only known-noise checks and say which: Trivy onCVE-2026-14456(base-image openssl) is the current one — confirm the CVE id in the log before dismissing it. - Record the final
origin/mainSHA in the doc header. That SHA is the base of the sweep; anything merged after it is a follow-up entry, not silently absorbed.
Phase 1 — Stack at main (not the feature branch)
pnpm wt-stack up builds from the worktree's checkout. A sweep on a
feature branch verifies the wrong code (bit us 2026-08-24, 47 commits stale).
git fetch --deepen=200 origin main # herdr worktrees are shallow
git checkout -b qa/sweep-post-<from-tag> origin/main # fixes land here
sed -i '' 's|^BREEZE_DOCKER_SUBNET=.*|BREEZE_DOCKER_SUBNET=<fresh /24>|; s|^BREEZE_CADDY_IP=.*|BREEZE_CADDY_IP=<same /24>.10|' .env
source ~/.nvm/nvm.sh && nvm use # .nvmrc pin, not host node
pnpm install --frozen-lockfile # code-mounted node_modules — stale tree = Astro NoMatchingRenderer
pnpm wt-stack up # read .breeze-stack.json for baseUrl + creds
Pick the /24 from docker network ls -q | xargs docker network inspect --format '{{.Name}} {{range .IPAM.Config}}{{.Subnet}}{{end}}' — an existing network with the same subnet fails compose with "Pool overlaps", running or not.
Boot with every feature flag the inventory (Phase 2) names, or the surface is
invisible and the check reads as a false FAIL. If the API is unhealthy with a
does not provide an export named … error, the dev image is stale — pnpm wt-stack up --rebuild. The seed has one org and a non-platform-admin; promote
it for /admin/* rows (update users set is_platform_admin=true where email='admin@breeze.local') and let a sweep agent create sibling orgs. Then
log in yourself once before dispatching any browser agent; a blocked sweep
agent is 15 wasted minutes.
Stack traps seen 2026-09-08 (v0.110→main sweep): (1) the first wt-stack up on a fresh DB fails with "api is unhealthy" because the healthcheck window closes during the ~645-migration replay — wait for Applied N migration(s) in docker logs <api> then run up again, it is idempotent; (2) set WEBAUTHN_RP_ID=localhost in the sweep .env or every passkey ceremony throws SecurityError (rp.id comes back as the prod domain) — the CDP virtual authenticator does not help; (3) SMS has no local provider (POST /auth/phone/verify → 501), phone-MFA rows are BLOCKED by design, say so; (4) role-forced MFA (force_mfa on Partner Admin) can lock the seeded admin into /auth/mfa/setup?forced=1 — MFA_FORCE_FOR_PARTNER_ADMIN=false is the relief valve (default OFF since #5307); (5) the seed's Invite offers partner-scoped roles only, so an org/site-restricted reader must be created via SQL; (6) fixer first-passes regress: run one review round per fix even when the diff is small — two of nine fixes this sweep needed it.
Stack traps seen 2026-09-29 (v0.118.2→main sweep): (1) a fresh worktree .env copied from .env.example has FORCE_HTTPS=true, NODE_ENV=production and a placeholder PUBLIC_API_URL, and the dev override defaults PUBLIC_API_URL to the prod domain — every API POST returns 400 until you set FORCE_HTTPS=false, TRUST_PROXY_HEADERS=false, and PUBLIC_API_URL / PUBLIC_APP_URL / DASHBOARD_URL / CORS_ALLOWED_ORIGINS to the local base URL; (2) the dev override defaults BREEZE_WORKSPACE_ENABLED to true — set it false on a stack without pgvector; (3) when the Playwright MCP browser is held by another session, drive headless Playwright scripts instead: save storageState after every script (refresh tokens rotate, and a script that dies before saving leaves the next run logged out) and set localStorage['breeze-onboarding-complete']='true' to skip the onboarding overlay.
Phase 2 — Change inventory (sonnet, no browser)
Dispatch one Explore agent (model: sonnet) to produce the inventory table:
merged PRs since the tag, surface (web / portal / api-only /
agent-only / docs-chore), a concrete click-path read from source
(real route from apps/web/src/pages, real button labels), the visible
outcome that proves the change, and prereqs (flag, role, live agent) that make
it BLOCKED locally. It must also list every BREEZE_*/*_ENABLED env var in
the diffs and cluster the click-paths into 4–6 area groups.
Paste the table into the doc with a Status column = TODO. Every row ends
the sweep as PASS | PARTIAL | FAIL | BLOCKED | N/A — never left TODO.
Phase 3 — Browser sweep (opus, one group at a time)
One agent owns the Playwright MCP browser at a time; groups run
sequentially. Each agent gets: the stack descriptor, its group's rows
verbatim, the doc path, and the rules below. It appends its rows' results plus
a "Paper cuts" sub-list before returning. Read ui-qa-sweep Phase 3/4 for the
everyday-workflow checks to fold in when a group is thin.
Rules every sweep agent must follow (they are the recurring false-positive sources):
- Toast host is custom:
[data-testid="toast"], auto-dismiss 5 s. Click and poll for the toast inside onebrowser_evaluate, or install a MutationObserver recorder before clicking. A "no toast" reading is a measurement, not a finding, untilgrep -n runAction <component>shows nosuccessMessage. - Scope element lookups to
mainor the drawer — tab labels collide with sidebar links by text. - Modals are
fixed inset-0divs with norole=dialog; assert on title text ordata-testid, and on the outcome, not the container. - A 2xx with no visible confirmation is a FAIL, not a PASS.
- Known noise, note once:
428on the firstPOST /auth/login(session-binding handshake, the retry succeeds); non-platform-admin 403s on/admin/*. - The API limiter is 300 req / 60 s per client. A fast nav crawl trips it and every page then "fails" with 429 — pace the crawl; a burst of 429s is the agent's own doing, not a finding.
- Fix agents run concurrently with the sweep on a code-mounted stack: a web
edit hot-reloads one page; an
apps/api/srcedit restarts the API for ~40 s. Tell the sweep agent which files are being edited and to treat a Vite overlay / 502 burst there as "wait 20 s and reload"; run API-side fixes in an isolated worktree (isolation: "worktree") and cherry-pick after the sweep group finishes.
Phase 4 — Fix small, file the rest
"Small" = one file or one component, no tenancy/auth/migration/billing/agent
surface, reproducible in a unit test. Fix on the qa/sweep-* branch, red
test first, then npx vitest run <file> (no -- before the flag; never a
trailing-slash path). Dispatch a sonnet agent per fix with the exact
repro + file. Tell it: lint/typecheck in the foreground with timeout 240
(a backgrounded run stalls the agent with the fix uncommitted — SendMessage
it to finish if that happens); --pool=threads --maxWorkers=2 if vitest can't
start workers under the running stack; never an eslint-disable for a rule
not in the package's eslint config (the comment itself is the lint error).
Verify its commit exists, skim the diff, and run eslint on the changed files
yourself before recording it. Run the heavy web tsc --noEmit once from the
main session, not per agent.
Anything larger → GitHub issue via github-issues, dedupe first, cite the doc.
Both go in the doc: Fixes applied (commit SHA) and Issues filed (#).
Phase 5 — Close
Summary table (group → PASS/PARTIAL/FAIL/BLOCKED counts), "Top findings"
(systemic patterns beat single bugs), the oldest PR reached, and what a
release cut still needs (flags to enable, BLOCKED rows needing a live agent).
Open one PR: the fixes + the tracking doc. Then tear down: pnpm wt-stack down
from this worktree, and docker compose ls -a to confirm nothing from the sweep
is still up — sweep agents' pnpm test-stack copies included (worktree-stack
skill → "Tear down when done"). Report anything you left running and why.
Model tiering
| Work | Agent |
|---|---|
| Inventory, env-var grep, mechanical single-file fixes | sonnet |
| Browser sweep (judgement on paper cuts, false-positive traps) | opus |
| Review of a fix that touched auth/tenancy | opus, one round |
Red flags
- Sweeping on the branch you happened to be on → wrong base, re-do Phase 1.
- A row marked FAIL with no API status+body captured → not a finding yet.
- "Merged, didn't check head-SHA CI" → the sweep merged red (#4159).
- Tracking doc written at the end → context loss ate the middle of the sweep.
- A "no toast → silent failure" finding whose click and poll were in separate
tool calls → re-measure in one
browser_evaluatebefore filing. - A row marked PASS on a stack whose flags were off for that surface → the surface never rendered; check the flag list in the doc header.
Stack traps seen 2026-10-23 (v0.124.0→main sweep): (1) the login limiter is 10 attempts / 5 min per IP and every client (curl, each sweep agent, your own browser) shares caddy's IP; a UI login costs 2 attempts (428 handshake + real) — log in once per agent, save storageState, never replay another agent's session (refresh-token rotation trips reuse detection); clear login:ip:* in redis only from the orchestrator; (2) when the MCP browser is held, headless agents can run two groups in parallel — they then share the 300 req/60 s limiter too, so brief both to pace ≤ 2 req/s and treat 429 bursts as noise; (3) merging origin/main into the running code-mounted stack mid-sweep: the API tsx watch can restart-loop for minutes and web/portal keep serving stale packages/shared types — docker compose … restart api web portal, then docker exec -w /app/apps/api <api> npx tsx scripts/migrate.ts for any new migrations (auto-migrate runs only at container start); (4) fixer worktrees branch from origin/<qa-branch> — push the qa branch before dispatching or they base on the wrong commit; (5) a fixer with a cd fallback edited the main checkout once — tell fixers "work ONLY inside this worktree".
版本历史
-
0b71144
当前 2026-11-03 11:01
更新v0.124.0到main的预发布Sweep结果,记录44个修复项、UI/UX问题及新发现的Issue,并同步了共享登录预算和并行组等陷阱经验。
-
8f240dd
2026-10-19 17:56
新增Phase 0中忽略Trivy CVE-2026-14456噪音检查的说明;更新Phase 1栈启动陷阱记录(v0.110→main sweep期间的健康检查窗口问题)。
- 1c3a08e 2026-09-22 22:17


