Supercheck monitoring and alerts

flowchart LR
  SCHED[Distributed monitor scheduler] --> Q[Regional queues]
  Q --> CHECK[HTTP, website, ping, port, or synthetic execution]
  CHECK --> AGG[Regional aggregation]
  AGG --> STATE{Alert state machine}
  STATE --> HIST[Alert history]
  STATE --> NOTIFY[Notification providers]

Start here

Monitor definition and execution

• Monitor definitions are tenant-owned configuration; executions and regional results are immutable observations.
• Preserve target, type, request/check config (method, headers/body, auth, timeouts, expected status/keywords, SSL checks), synthetic test reference, frequency, location selection, alert thresholds/flags, muted state, enabled state, and notification relationships across schema, APIs, UI, scheduler, queue, and worker. Monitor retryStrategy is deprecated and ignored.
• Monitor destinations are untrusted. Validate URL shape at configuration time; enforce SSRF-safe DNS-pinned resolution at execution time, re-applied on every redirect and against DNS changes.
• Scheduling is safe across multiple app replicas via BullMQ repeatable monitor jobs keyed by a stable jobId, with a Redis NX lock guarding startup re-scheduling (app/src/lib/monitor-scheduler.ts). Restarts or overlapping ticks must not duplicate dispatch.
• Location selection maps deterministically to worker queues and rejects unknown/unavailable locations deliberately.

Aggregation

• A regional result and aggregate monitor state are separate. Never infer global health from one region.
• Define how missing, late, timed-out, and conflicting regional results affect aggregate status; preserve that rule across dashboards, APIs, and alert evaluation. Cycle status is open, closed, or aborted (app/src/db/schema/monitor-cycles.ts), and the work is split:
  • decideMonitorCycle in worker/src/monitor-control/cycle-policy.ts computes the regional quorum decision only.
  • MonitorCycleService.closeCycle (monitor-cycle.service.ts) applies it: monitor status, synthetic missing-region results, and alert evaluation.
  • monitor-cycle.service.ts aborts cycles for paused or disabled monitors. An aborted cycle writes no synthetic failures, so it does not count against uptime.
  • monitor-cycle.processor.ts backs off and retries closure when a dependency check is temporarily unavailable, instead of failing the cycle.
  • recordCycleArrival in worker/src/monitor/monitor.service.ts treats a cycle owned by another monitor as absent, and a result for a closed or aborted cycle never re-evaluates it.
• Use consistent units and boundaries for latency, uptime, thresholds, and time windows.
• Store enough immutable evidence to explain why an alert transitioned without retaining secrets or unrestricted response bodies.

Alert state and history

• Model transitions explicitly using the current states and consecutive-failure/recovery policy.
• Persist state transitions with the result that caused them; alert_history records each delivery attempt (provider, status, outcome).
• Deduplicate repeated failing observations while preserving meaningful reminders/escalations configured by the product.
• Recovery closes the active condition once (one recovery alert per outage); failure reminders back off exponentially and are capped at 3 per sequence.
• Alerts open incidents: a sent or failed (not pending) alert_history row is queued for app/src/sre/lib/background-alert-triage.ts, which opens or joins one incident per monitor or job. alertConfig carries incidentEnabled (default on), incidentSeverity (auto or sev1-sev4) and autoResolveIncident (default on); a recovery resolves the open auto-opened incident when it is on, even if recovery alerts are off. Only SRE_ENABLED is needed; SRE_AUTOMATION_ENABLED gates AI triage. Keep these fields in sync across the app schema, worker, CLI schema, MCP tools and OpenAPI.
• Alert acknowledgement is not implemented; incident acknowledgement is (see the SRE operations skill). Manual status changes require permission, tenant scope, audit events, and race-safe updates.

Notification providers

• Provider definitions (email, Slack, webhook, Telegram, Discord, Teams; webhook presets for PagerDuty, Opsgenie, Splunk On-Call, Better Stack, and incident.io) are scoped to their owning tenant and secrets remain encrypted/redacted.
• Render messages from bounded, escaped data. Do not leak headers, variables, tokens, raw response bodies, or internal errors.
• Delivery uses bounded timeouts with provider-appropriate retry behavior (Slack, Telegram, Discord, and Teams retry; webhook and email are single-attempt today).
• Webhook destinations use SSRF protection (allowlisted hosts, DNS pinning). Notification delivery is strictly outbound; no inbound callback route exists.
• Record delivery outcome and safe provider diagnostics without blocking the monitor’s terminal persistence indefinitely.

Verify

• Unit-test threshold boundaries, state transitions, deduplication, recovery, and partial/missing region behavior.
• Integration-test scheduler ownership, queue routing, worker output, and provider retry/idempotency.
• Test cross-tenant provider/monitor access and secret redaction.
• Use disposable provider endpoints and clean them up; live delivery acceptance remains separate from automated tests.