Designing cache keys
A key must uniquely locate one piece of data and change with its dependencies. A common bug is omitting version or tenant from the key, mixing data across scopes.
TTL is not "the bigger the better"
Longer TTL risks stale reads; shorter hurts hit rate. Set by mutability: config can be long, inventory must be short or actively invalidated.
Three failure modes
| Problem | Symptom | Fix |
|---|---|---|
| Penetration | Queries for missing keys hit DB | Cache nulls / bloom filter |
| Stampede | Spike when a hot key expires | Mutex rebuild / logical expiry |
| Avalanche | Many keys expire together | Add jitter to TTL |
Update or delete
After a write, prefer deleting the cache (lazy rebuild); it is simpler and less inconsistent than write-back.
Real-world cases: three cache-induced incidents
- Old price after an update: the write succeeded but the cache was not invalidated. Under concurrency both "delete then write" and "write then delete" can serve stale reads. The steadier rule is write the DB first, then delete the cache, with a brief mutex on hot reads.
- Caching nulls hides the fix: to stop penetration you cache empty results for 10 minutes — then after the upstream recovers users still see "no data". Give nulls a shorter TTL, or expose a manual purge.
- Multi-level caches disagreeing: an in-process cache and Redis fall out of sync, so one user sees two different answers on refresh. Use a very short local TTL, broadcast invalidation, or drop to a single level.
FAQ
Why not delete the cache before writing the DB? A read arriving between the delete and the write reloads the old value, creating long-lived stale data. Why add jitter to TTL? To stop many keys expiring in the same second and causing an avalanche. Should caches hold large objects? No — serialization and network cost are high and single requests slow down; shard them or use object storage. How do I verify invalidation? Write a test that writes then immediately reads and asserts the new value, then run a concurrency load test to check for regressions.
Make invalidation a verifiable process
Cache incidents usually stem from a missing process rather than a wrong technology choice. These practices cut both staleness and avalanche risk.
- Name the authority for every piece of data: decide which system is the single source of truth and treat the cache as its projection. Two writable stores leave consistency to luck, so writes go only to the authority and the cache updates passively.
- Design keys for bulk invalidation: organise keys as object plus dimension plus version, with tenant and type in the prefix, so a class of data can be deleted by prefix instead of enumerating every key. Being unable to invalidate in bulk is a common amplifier of cache incidents.
- Detect invalidation failures: a delete can fail on a network blip. Count failures, alert past a threshold, and keep fallbacks such as temporarily bypassing the cache for that key.
- Give hot keys their own policy: when a heavily read key expires, rebuild pressure lands on the database. Logical expiry — keeping a stale marker and letting one bounded worker rebuild while others serve the old value — absorbs that.
- Load-test the invalidation path: before release, simulate mass expiry and a brief database outage, and confirm the system degrades rather than erroring. This check catches the worst failures early.
Align with business semantics
Technical TTLs must match the staleness the business accepts: inventory is often seconds, article bodies tolerate minutes. Write those requirements into the API description and TTLs stop being guesswork.