Incident first response

The seven steps

1. Detect. Notice the incident — via monitoring alert, error spike, or user report. This is why monitoring must alert a human: detection delayed is resolution delayed.
2. Acknowledge. State it plainly: "We have an incident: [one-line description]. I'm on it." If you're a team, the on-call acknowledges in the shared channel so everyone knows it's owned. Unacknowledged incidents get "fixed" by three people at once.
3. Assess. Before touching anything, answer three questions: What is broken? Who is affected? How bad is it? Assign a severity (below). Five minutes of assessment saves thirty minutes of wrong-direction fixing.
4. Communicate. Post the first outward update early — even if it's just "We're investigating an issue with [thing]." Users tolerate incidents; they don't tolerate silence. Then update on a rhythm (every 30 minutes for S1, hourly for S2) until resolved.
5. Mitigate. Stop the bleeding first, fix the root cause second. Mitigation is the rollback, the feature flag flip, the traffic shift, the maintenance page — whatever restores service fastest. Root-cause fixes come after users are unblocked.
6. Resolve. Confirm the fix the way you'd confirm a launch: the failing thing now works, error rates are back to baseline, conversions are flowing. Then declare the incident resolved, publicly.
7. Review. Within 48 hours, run the blameless postmortem (template below). The incident isn't over when service is restored — it's over when the lesson is captured.

Severity levels

Severity	Definition	Response
S1	Site down, checkout/signup broken, data loss or corruption, security exposure	Drop everything. Update every 30 min. Consider rollback immediately.
S2	Major feature broken with a workaround, degraded performance, elevated errors on a key path	Fix today. Update hourly. Rollback on the table.
S3	Minor, cosmetic, single-user edge case	Log it, schedule it. Not an incident — a ticket.

If you're unsure between S1 and S2, call it S1. Downgrading later is free; under-responding to an S1 is expensive.

Roles

On a team, assign these at the start of the incident — one person can hold multiple roles, but every role must be held:

• Incident commander: owns the response, makes the calls (including rollback), keeps the response moving. Does not also debug — commanders who go heads-down lose the picture.
• Responder: investigates and implements the fix or mitigation.
• Communicator: owns every outward message — status updates, support replies, social posts. One voice, no contradictions, no speculation presented as fact.

Solo founder? You're all three. The discipline still applies: before you start debugging, write the one-line acknowledgment (even if only in your notes), and post the first user-facing update before you're done fixing.

Communication templates

Don't compose under pressure — adapt these:

• Investigating: "We're investigating reports of [symptom] affecting [scope]. We'll update here within 30 minutes."
• Identified: "We've identified the cause: [plain-language cause]. We're [mitigation step] now. Next update in 30 minutes."
• Monitoring: "Service is restored. We're monitoring closely and will confirm stability before closing this out."
• Resolved: "Resolved at [time]. [One-line cause]. We'll publish a summary of what happened and what we're changing within 48 hours."

Rules: plain language, no jargon, no blame, no speculation about causes you haven't confirmed, and never "a small number of users" when you don't know the number.

The blameless postmortem

Within 48 hours. Written. Blameless by rule — the question is always "how did the system allow this," never "who did this." Template:

1. Summary: what happened, in two sentences.
2. Timeline: detected at, acknowledged at, severity assigned, mitigation started, resolved at. Times, not narratives.
3. Root cause: the technical cause and the process cause. (The deploy broke checkout and the smoke test didn't cover checkout.)
4. What went well: detection speed, the rollback runbook working, communication rhythm — reinforce these.
5. What went badly: be specific and mechanical.
6. Action items: each with an owner and a date. A postmortem without action items is a diary entry.

Prepare before you need it

• Monitoring that alerts a human, with the on-call path written down (this is a scored assessment item for a reason).
• A rollback runbook that's been rehearsed — mitigation is often just a rollback.
• A place to post status updates, decided in advance (status page, social account, in-app banner).
• Access: make sure the responder can actually reach production systems at 2 a.m. — credentials, VPN, permissions, all tested.