Rollback runbook

What qualifies as a rollback

Not every bug deserves a rollback. Define the triggers in advance so the decision is recognition, not debate:

• Roll back: site down or unreachable, checkout/signup completely broken, data corruption or loss, security exposure introduced by the deploy.
• Fix forward: cosmetic issues, copy errors, a single non-critical feature misbehaving, analytics misfiring. These get a follow-up deploy, not a rollback.
• Gray zone: degraded performance, elevated error rates on a non-critical path. Decide in the moment using one question: are users better served by the previous version right now? If yes, roll back.

Who decides

Name one person with rollback authority before launch. On a solo team, that's you — which means writing down the triggers above so future-you, mid-adrenaline, follows the plan instead of improvising. On a team, it's whoever is on call, with no committee required. Rollback authority must be usable in under two minutes, including at 2 a.m.

What "ready to roll back" actually requires

• The previous artifact is retained. The exact build that was live before — container image tag, deployment bundle, or platform release — kept available, not overwritten. Know its identifier (tag, version number, deploy ID) in advance.
• A pre-deploy snapshot exists. Database snapshot, content backup, or volume snapshot taken immediately before the deploy. Know where it lives and how long a restore takes — you measured this in the preflight restore drill.
• The procedure is written step by step. The exact commands or click-path, in order, including how to verify each step. "Roll back the deploy" is not a procedure; twelve numbered steps are.
• Database migrations are rollback-safe. This is where rollbacks actually die. Follow the expand/contract pattern: migrations should only add (new tables, new nullable columns) — never rename or drop in the same deploy that starts using the change. A rollback of application code against a database that has already dropped the old columns is a second outage. If a deploy includes a destructive migration, the rollback plan must include the corresponding database restore, tested, with its duration known.

The procedure

1. Declare it. State clearly (to yourself, your team, your status channel): "We are rolling back to [artifact] because [trigger]." This prevents two people fixing forward and rolling back simultaneously.
2. Stop the bleeding first if it's data-related. If the deploy is corrupting data, halt writes or put the app in maintenance mode before rolling back — restoring code over a still-corrupting database just repeats the damage.
3. Restore the previous application artifact using the written steps. Don't improvise; follow the runbook.
4. Restore the database snapshot only if required (schema changed destructively, or data was corrupted). Database restores lose everything written since the snapshot — weigh that cost explicitly before doing it.
5. Verify like it's a fresh launch: homepage loads, login works, checkout/signup works, no elevated errors in the logs. The rollback isn't done when the deploy finishes; it's done when the site works.
6. Communicate. Update the status page or announcement channel: what happened, that service is restored, what you're doing next. Even a brief note beats silence.

After: the post-rollback review

Within 48 hours, while memory is fresh, write down:

• What triggered the rollback, and was the trigger list correct?
• How long did detection, decision, and recovery each take?
• Did the runbook work as written? What step was missing or wrong?
• What change to the deploy process would have prevented this — smaller deploys, a staging gate, a migration review?

Blameless by rule: the review is about the system, not the person. A rollback that followed the runbook is a success of the runbook, not a failure of the deployer.

Rehearse it

A runbook you've never executed is a draft. During the 48-hour preflight, roll staging back to the previous build and time the whole procedure. Update the runbook with everything you learn. The rehearsal is also when you discover the missing permission, the expired credential, or the step that only works from one person's laptop.