Why not all at once
A buggy new version pushed to everyone hits all users instantly, with no time to roll back. Progressive releases shrink the risk so a problem touches only a few.
Canary release
Roll out by traffic percentage: 1% internal first, then 5%, 20%, 100% as metrics stay healthy. Stop and revert at any anomaly. Good for real-traffic validation.
Blue-green deployment
Run two environments: blue (old) serves traffic while green (new) is ready; flip routing to green in a second. Flip back to blue on trouble, users barely notice. Cost is double resources.
Rollback matters more than ship
- Keep the previous version: blue-green has a backup by design; canary must scale back to 0% in one click;
- Watch metrics before scaling: stop on errors, latency or conversion spikes;
- Backward-compatible schema: new code reads the old table, so rollback will not crash.
Real-world cases: three failed releases
- Connections cut during cutover: removing the old environment outright kills long connections and in-flight requests. Stop new traffic first, drain, then retire the old environment.
- Canary by machine instead of by user: one user bounces between instances and sees both old and new UI, or inconsistent state. Route canary traffic consistently by user or request identity.
- Irreversible database change: code can roll back but the column was already dropped, so the rollback fails instantly. Use expand-and-contract: add the column and dual-write, then remove the old one once stable.
FAQ
Blue-green vs canary? Blue-green swaps two complete environments — fastest rollback, double the resources; canary exposes a small slice first — leaner and more observable. What canary percentage? Start at 1–5%, watch error rate and latency, then widen step by step. How do I roll back fast? Keep artifacts traceable to a version and make database changes backward compatible. Do I need feature flags? Yes — they disable new features without redeploying, a second gate on release risk.
Preparation before and validation after
A release strategy exists to make recovery fast; what decides success is the preparation around it.
- Observability before rollout: canaries need metrics, at minimum success rate, latency percentiles and key business indicators broken down by version. A canary you cannot measure is not a canary.
- A change list and rollback point: record what changed, which data changes it depends on, and what blocks a rollback. Database and configuration changes are usually harder to reverse than code — flag them separately.
- Automated validation: compare core metrics between versions during the canary and halt automatically past a threshold. Humans watching a screen miss signals at the worst moment.
- Rehearse the rollback: practise it before the incident, not during it. Drills surface incompatible data, unsynced configuration and unarchived images.
- Window and environment isolation: avoid peak hours, keep canary traffic controllable and abortable, and queue changes in shared environments so they do not mask each other.
Working with team process
Beyond tooling, cadence and ownership matter: decide in advance who halts the rollout, who rolls back and who is informed. Giving one clear person the call beats a meeting after the fact.
A minimum viable approach for small teams
Not every team needs a full canary system. With limited resources, do at least three things: traceable artefacts with one-click rollback, core metric comparison before and after, and version-controlled changes with configuration. That alone recovers most release incidents in minutes.