The problem
When a downstream slows down, callers wait and retry without end; thread pools fill up and the failure climbs the call chain until the whole site avalanches.
Three breaker states
| State | Behaviour |
|---|---|
| Closed | Normal pass, count failure rate |
| Open | Fail fast, stop hitting downstream |
| Half-open | Let a few requests probe recovery |
Above a failure threshold it opens; after a cool-down it goes half-open to test; success returns to closed, failure stays open.
What degradation is
Instead of blocking, serve a fallback: a default list for recommendations, cached data for a minor API, comments temporarily off. Keep the main flow alive and sacrifice the extras.
Two reminders
- Cap timeouts: breaking needs a sane timeout per call, or you hang in pre-open waiting forever;
- Fallbacks must be correct: degraded data should not mislead users or trigger a second error.
Real-world cases: three times the circuit breaker made things worse
- Breaking without degrading: once the dependency fails, requests are rejected and the page shows nothing. Pair breaking with degradation: cached data, defaults or a clear message.
- Thresholds too twitchy: a few timeouts trip the breaker, so it flaps on normal jitter. Combine error rate with a minimum request count and a time window, and use half-open probes to recover gradually.
- Cascading timeouts that never converge: a 30-second upstream timeout with three retries per layer turns one failure into an avalanche. Keep each layer's timeout below its caller's and retry only retryable errors.
FAQ
Breaking vs rate limiting? Rate limiting protects you from being overwhelmed; breaking avoids hammering a dependency that is already down. When does it recover? In half-open state a few probes go through; once they succeed at the target rate it closes fully. What should degradation return? Acceptable stale or default data, clearly marked as temporarily unavailable — never silently empty. Break every dependency? Only strong dependencies that can be down for long; adding it to fast internal calls just adds complexity.
Designing degradation that actually helps
Breaking stops the call; what users experience is the fallback. Follow these principles.
- Tier by data sensitivity: core data should not degrade — fail clearly instead; recommendations, statistics and display-only data can return cached or empty values so the main flow survives.
- Make degradation visible: show that data may be stale rather than silently serving old content, or users make decisions on wrong information.
- Keep manual switches: automatic breaking can misfire, so provide a fast human override, scoped per dependency rather than global.
- Test the fallback path: rarely used branches fail exactly when needed — include degradation switches in regular tests and rehearse them.
- Measure duration and frequency: report degradation events as metrics; frequent long degradations mean the dependency itself is unstable and needs fixing.
Pair it with capacity policy
Breaking handles an unavailable dependency; rate limiting protects you from being overwhelmed. Use both — breaking alone lets bursts exhaust your own resources, and limiting alone keeps hammering a dead dependency.
Priorities with many dependencies
When several dependencies fail at once, decide by business value which capabilities survive: protect checkout and payment first, degrade recommendations and analytics. Define the priority in advance so nobody argues during the incident — and so the degradation logic stays simple.