Systems administration

Well-run systems are boring. The work is in the routine that keeps them that way, and almost every serious incident traces back to a routine that was skipped.

Cloud environment design belongs to technology:cloud-infrastructure; this is operating the systems the company runs on.

Configuration baselines and drift

Every system class needs a defined baseline: build, hardening, agents, logging, accounts. Systems built by hand from memory diverge immediately and cannot be reasoned about as a group.

Drift is the real enemy. Detect it continuously and correct rather than document — a system that no longer matches its baseline is a system whose behavior under patching or failover is unknown.

The strongest form is disposability: rebuild rather than repair. A system you can rebuild in an hour never accumulates the sediment of a decade of manual fixes.

Patching as a cadence

Set a regular, predictable window and hold it. Ad hoc patching means patching happens when someone worries, which is never in proportion to actual risk.

Tier by exposure: internet-facing systems on the shortest cycle, then internal, then isolated. Emergency patching is a separate path with its own authority, used for actively exploited vulnerabilities — security:vulnerability-management decides what is urgent, this skill executes it.

Track coverage, not activity. "Patching is running" is not an answer; "97% of servers are within 30 days, here are the twelve that are not and why" is.

Change control proportionate to risk

Heavyweight approval for trivial changes produces circumvention, and circumvention produces unrecorded changes, which is worse than no process.

Tier it: standard pre-approved changes, normal changes with review, emergency changes with after-the-fact record. Every change needs an owner, a back-out plan, and a record — the back-out plan being the part most often assumed rather than written.

Maintenance windows exist to make disruption predictable. Announce them, keep them, and finish inside them.

Know what you have

An unmanaged system is a system nobody patches. Reconcile what is running against it-operations:it-asset-management regularly, and treat anything unaccounted for as urgent — the server nobody owns is the one still running an unsupported operating system.

Sources

references/sources.md in this skill lists the outside authorities that settle the questions here — what each one is authoritative for, and what you may do with it. Check them before answering on anything they cover, and cite what you used. Most are free to read and not free to reproduce; the use note on each is binding.

Never

• Repair a drifted system without correcting the baseline that let it drift.
• Patch on worry rather than cadence.
• Make a change with no back-out plan.
• Leave a discovered unmanaged system unclaimed.