Supercheck SRE operations

AI SRE architecture

flowchart LR
  SIGNAL[Monitor and telemetry signals] --> INCIDENT[Incident]
  INCIDENT --> ORCH[AI SRE orchestrator]
  ORCH --> AGENT[Specialized agents]
  AGENT --> TOOLS[Scoped evidence tools]
  TOOLS --> BRIEF[Evidence and diagnosis]
  BRIEF --> HUMAN{Authorization gate}
  HUMAN -->|approved| MUTATE[Bounded action]

• AI SRE source is split between app/src/sre, app/src/lib/sre, API routes, components, and matching worker schema/types.
• Incidents, services, connectors, private agents, conversations, messages, evidence, tool calls, and usage remain scoped to their owning organization/project.
• Authorization is enforced server-side for every tool call and persisted resource. Model instructions and UI visibility are not authorization.
• SRE_ENABLED (default on) alone makes monitor and job alerts open incidents; SRE_ENABLED=false disables all SRE work. SRE_AUTOMATION_ENABLED=true|1 gates only the AI triage step on top of that, and only the app reads it; the worker reads SRE_ENABLED only.
• Prefer read-only evidence gathering. Mutations require explicit user authority, least-privilege credentials, bounded targets, audit records, and observable outcomes.
• Kubernetes access uses namespace-scoped service accounts and allowed resources/actions; never grant cluster-admin.
• Preserve stable assistant/user message identity through streaming, persistence, polling/reconciliation, retries, and reloads.
• Incident lifecycle (assign, unassign, acknowledge, commander, resolve) is in app/src/actions/sre-incident-lifecycle.ts and /api/sre/incidents/{id}/assign|acknowledge|resolve, all gated by sre_incident:update. Resolve needs a summary (max 2000 characters); assignees must be org owner/admin or project admin/editor. Incident notifications use a fixed body and never include AI text or raw evidence.
• Alert to incident: the worker queues each sent or failed (not pending) alert; app/src/sre/lib/background-alert-triage.ts opens one incident per source (monitor or job), joins later failures to it via sre_incident_alerts, and titles it <target>: <alert type> (app/src/lib/sre/alert-labels.ts). Per check, alertConfig has incidentEnabled (default on), incidentSeverity (auto or sev1-sev4) and autoResolveIncident (default on).
• Auto-resolve: when a check recovers and autoResolveIncident is on, an unacknowledged auto-opened incident resolves even if recovery alerts are off or no channel is notified; an acknowledged one stays open with a note. A new failure within 30 minutes (INCIDENT_REOPEN_WINDOW_MS in app/src/lib/sre/incident-lifecycle.ts) reopens it.
• Incident emails go to participants (assignee, commander, and the previous assignee on reassign) through notifyIncidentEvent in app/src/lib/sre/incident-notifier.ts: never to the actor, only to users listAssignableUsers still returns, and recovered/reopened at most once per person per 10 minutes. Channels skip recovered and automatic resolved/reopened, because the worker already posted them. Each sre_incident_alerts row snapshots the alert (alert_title, alert_description, alert_severity, alert_fired_at, migration 0046); read it through linkedAlertFields in app/src/lib/sre/incident-alert-snapshot.ts, because the shared sre_alert_events row is refreshed by later failures.
• Six display statuses (app/src/lib/sre/incident-status-display.ts: Triggered, Acknowledged, Investigating, Fixing, Monitoring, Resolved) map from the seven stored statuses. The API, MCP and CLI return statusLabel; status filters take stored values.
• Investigations are evidence-first: they read stored native evidence (monitors, runs, alerts) before any live connector, and live sources are opt-in per run. Token use from investigations, triage, briefs and Copilot counts against the daily org cap (see Environment configuration).
• Connector and private-agent credentials stay encrypted/server-side and are never placed in prompts, browser payloads, logs, or model-visible errors.

Tool and provider behavior

• Tool schemas validate inputs and bound output size. Treat provider/tool output as untrusted evidence.
• Jira, Confluence, and Notion are read-only knowledge connectors (Notion is direct only; Jira and Confluence also run on a Private Agent in worker/src/private-agent/private-agent-knowledge.ts). Page text is redacted, size-capped, fenced as untrusted document evidence, and fetched for top results only. Slack, Teams, Webhook, Kafka, and RabbitMQ are not evidence sources; unsupported records show as Unsupported and must be disableable.
• Cancellation and timeout propagate through model streams, tools, polling, persistence, and UI state.
• MCP connectors are Private Agent only, Streamable HTTP only, never stdio. Only tools allowlisted by a user with sre_connector:configure can run (description and schema are pinned at allowlist time); arguments are validated against the saved schema in app/src/sre/tools/mcp-tools.ts and again in worker/src/private-agent/mcp-client.ts. Output is text/JSON only, redacted, and untrusted; server read-only hints never authorize a call. The agent must run the same release as the app.
• Private Agents report capabilities.sreConnectorTypes (worker SUPPORTED_SRE_CONNECTOR_TYPES), stored in private_agents.metadata; the job router refuses types an agent did not report (private_agent_outdated), and agents that report nothing are treated as supporting only the 10 original types. Keep the worker list equal to PRIVATE_AGENT_CONNECTOR_TYPES (a spec enforces it). Workers fail unknown-type jobs with unsupported_connector_type instead of re-leasing them.
• Retry only transient/idempotent operations; prevent duplicate messages, actions, tool records, usage events, and charges.
• Observability providers (Jaeger, New Relic, Splunk, Google Cloud, Azure Monitor) live in app/src/lib/sre/connectors/providers/ and are copied to worker/src/private-agent/providers/; edit the app files and run npm run sync:sre-providers in app/. Queries are read-only (reject Splunk write commands, non-SELECT NRQL, KQL control commands) and always time-windowed. New Relic, Google Cloud, and Azure call fixed vendor hosts.
• Provider fallback (including deterministic evidence-brief fallbacks) must preserve tenant scope, safety policy, configured model selection, usage accounting, and error semantics.
• Investigation results are structured and validated in app/src/sre/lib/investigation-output.ts (conclusion, confidence, ranked hypotheses; Inconclusive is valid; a hypothesis is Validated only with evidence from this incident). Similar past incidents (5 resolved, 180 days, same project) and user feedback notes are hints, never evidence. Post-mortem drafts are rate limited (5 per user per incident per hour) and fall back to a deterministic template. Changes under app/src/sre/** run the sre-eval CI gate.
• Redact secrets and sensitive telemetry before model invocation and persisted transcripts.

Resilience

• External calls have explicit connection/operation timeouts and abort handling.
• Use exponential backoff with jitter for safe transient retries; no circuit-breaker abstraction exists yet, so introduce one deliberately if sustained dependency failure needs containment.
• Fallbacks degrade explicitly and never bypass authentication, authorization, capacity, billing, or audit controls.
• Redis connections intentionally survive failover; QueueEvents blocking connections remain isolated from normal command timeout behavior.
• Distributed schedulers/locks use ownership tokens and bounded leases so one instance cannot release another’s lock.

Memory and performance

• Bound transcripts, evidence, logs, tool output, arrays, caches, queue payloads, and export/result sizes.
• Stream large responses/artifacts and paginate growing datasets.
• Clean up timers, listeners, subscriptions, AbortControllers, child processes, temporary files, and browser contexts on every terminal path.
• Optimize database access from measured query plans/cardinality; preserve exact React Query cache-key parity between prefetch and hooks.
• Do not trade tenant/security checks for caching or performance.

Environment configuration

• Environment access is mixed: the app reads process.env directly, and the worker combines @nestjs/config (ConfigModule.forRoot() plus injected ConfigService) with direct process.env reads in several services. No central schema helper exists today.
• Keep app, worker, deployment manifests, examples, docs, and health/config diagnostics synchronized. Self-hosting assets live in deploy/docker/ (compose variants, install.sh, init-secrets.sh, setup-k3s.sh, setup-gvisor.sh) and are tested by compose.spec.py and install.spec.py; the user guide is docs/content/docs/app/deployment/.
• AI SRE gates live in app/src/sre/lib/feature-gates.ts: SRE_ENABLED defaults on, SRE_AUTOMATION_ENABLED and SRE_AGENT_SANDBOX_ENABLED default off. The worker reads its own copy of SRE_ENABLED only (worker/src/monitor/services/sre-alert-triage-queue.service.ts); SRE_AUTOMATION_ENABLED is app-only. Keep the two SRE_ENABLED checks in sync.
• AI SRE HTTP routes authorize through app/src/app/api/sre/_auth.ts; dynamic [id] routes reject malformed UUIDs before database access.
• Daily AI token cap per organization: SRE_DAILY_TOKEN_LIMIT_PER_ORG (default 2000000, 0 disables, UTC day, needs Redis, app only) in app/src/lib/sre/sre-daily-token-limit.ts. Investigations, triage, briefs, and Copilot share it; keep it in app/.env.example, the self-hosted docs table, and the AI SRE setup page. Investigation runs stuck for 5 minutes are marked failed by stale-run-recovery.ts.
• Validate required production values and fail closed when security-critical settings are missing.
• Never expose secrets through health endpoints, startup logs, client-prefixed variables, build output, or configuration APIs.
• Use distinct credentials/key namespaces for auth, encryption, object storage, AI providers, billing, email, and infrastructure.
• BullMQ Redis requires noeviction; production execution requires gVisor/isolation settings.

Release management

flowchart LR
  CODE[Reviewed commit] --> CI[Required CI]
  CI --> TAG[Signed annotated release tag]
  TAG --> IMAGE[Immutable release images]
  IMAGE --> MIGRATE[Compatible migration]
  MIGRATE --> DEPLOY[Bounded deployment]
  DEPLOY --> ACCEPT[Exact-SHA acceptance]
  ACCEPT --> CLEAN[Fixture cleanup]

• Use semantic project versions and immutable image tags/digests; never deploy main, another branch name, or latest to production.
• Release tags must be annotated and signed with a key listed in .github/release-allowed-signers; unsigned, lightweight, or unlisted-key tags fail the publish gate. Agents never create, move, or delete tags or dispatch publish workflows (docs/releasing.md).
• Review database compatibility before rollout and order app/worker/migration changes for version skew.
• Migration 0043_sre_incident_management.sql (incident assignee, acknowledgement, commander, resolution summary, post-mortem columns) is forward-only and must be applied before the app that reads it is deployed.
• CI must not use pull_request_target with untrusted contributor code or expose secrets to fork builds.
• A successful build/deploy workflow is not release acceptance. Verify deployed SHA, health, migrations, browser flows, RBAC/flags, workers/control plane, provider/billing gates, and cleanup.
• Keep changelog, package versions, deployment defaults, docs, and release evidence synchronized.

Verify

• Cover tenant/RBAC denial before model/provider/tool execution.
• Cover cancellation, provider failure, transcript identity, retries, audit, usage/billing exactly-once behavior, and connector/private-agent boundaries.
• Use disposable non-production fixtures. Never inject failures into production, use customer credentials/data, exhaust limits, or spend quota only to repeat a proven scenario.