A few years ago I inherited a payments dashboard that showed merchants their settled balance. It was fast, users loved it, and roughly once a week someone in support would escalate a ticket that said "my money is wrong." It wasn't wrong. The cache was. Somewhere between a settlement job writing to the ledger and the dashboard reading from Redis, a stale value would survive a deploy, a retry, or a race, and a merchant would briefly believe we owed them 12,000 euros we did not.
Nobody had done anything stupid. Every individual caching decision was reasonable. The nightmare was invalidation, and invalidation is where caching quietly turns from a performance win into a correctness liability. I've spent a good chunk of my career cleaning up caches other people built with the best intentions, so let me tell you how I think about it now.
The problem was never the cache
Phil Karlton's line about there being only two hard things in computer science, cache invalidation and naming things, gets quoted so often it's become wallpaper. But it's true, and the reason it's true is subtle. Reading from a cache is trivial. Writing to a cache is trivial. Knowing the exact moment a cached value stopped being true, across every process that might have written the underlying data, is the hard part, because that knowledge lives in a different place from the cache itself.
Most stale-data bugs I've traced were not caching bugs in the mechanical sense. They were assumptions about who owns a piece of data and when it changes. The cache just made the wrong assumption visible three seconds later instead of immediately. So my first move on any caching design is not to pick a store or a TTL. It's to ask a blunt question: what is the source of truth for this value, and how many independent writers can touch it?
Reach for a TTL before you reach for invalidation
Here is the opinion that has saved me the most grief: an expiring cache with no explicit invalidation is almost always the right starting point, and most teams skip past it because it feels lazy. It isn't lazy. It's honest about the fact that your invalidation logic will eventually have a bug, and a TTL bounds the blast radius of that bug to its own duration.
If a merchant's settled balance is at most 30 seconds stale, and everyone downstream understands that contract, you have removed an entire class of production incidents. No cache-busting on write. No message to publish. No subscriber to get out of sync. The value just ages out. I would rather explain to a product manager that a number can lag by half a minute than explain to a merchant why our system claimed a payment that never landed.
The best invalidation strategy is the one you don't have to write. A short TTL is not a compromise; it's a design that fails closed instead of failing wrong.
TTLs are not free, obviously. They cost you a stampede risk when a popular key expires and forty requests all miss at once, and they cost you freshness you might genuinely need. But those are known, boundable problems with known fixes. Explicit invalidation across a distributed system is an open-ended problem, and I will trade an open-ended problem for a bounded one every single day.
Cache-aside, and why the write path is where it breaks
The most common pattern in the wild is cache-aside: on read, check the cache, fall back to the database, populate the cache. It's simple and it works. The trouble is what people bolt onto the write path. The tempting move is: write to the database, then update the cache with the new value. That double-write is a race waiting to happen. Two concurrent updates can land in the database in one order and in the cache in the opposite order, and now your cache disagrees with your database indefinitely, with no TTL to save you because you keep overwriting it.
My rule is to delete on write, never update on write. On a mutation, invalidate the key and let the next read repopulate it from the source of truth. It costs one extra cache miss. It also means the database, the one place with real ordering guarantees, always wins.
public async Task<MerchantBalance> GetBalanceAsync(Guid merchantId)
{
var key = $"balance:{merchantId}";
var cached = await _cache.GetAsync<MerchantBalance>(key);
if (cached is not null)
return cached;
var balance = await _ledger.GetBalanceAsync(merchantId);
// Short TTL: fails closed, not wrong.
await _cache.SetAsync(key, balance, TimeSpan.FromSeconds(30));
return balance;
}
public async Task RecordSettlementAsync(Settlement s)
{
await _ledger.ApplyAsync(s); // source of truth first
await _cache.RemoveAsync($"balance:{s.MerchantId}"); // then delete, don't update
}Notice the order: mutate the source of truth first, then invalidate. If you invalidate first and the database write fails, you've just done pointless work. If you invalidate first and a concurrent read repopulates the cache from the old database state before your write lands, you're stale again. Write, then delete. And if the delete fails, the TTL still catches you within 30 seconds. Belt and suspenders, on purpose.
Stampedes will find you at the worst moment
Once you lean on TTLs, you inherit the thundering herd. A key expires, and every in-flight request for it misses simultaneously and hammers your database at once. I've watched a perfectly healthy Postgres box hit 100% CPU because one very popular config key expired during a traffic spike and 2,000 requests all decided to recompute it in the same 40 milliseconds.
The fixes are well understood, and you should pick one deliberately rather than discovering the problem in production:
- Request coalescing: let the first miss recompute the value while other callers wait for that single result. In .NET a per-key
SemaphoreSlimor a shared lazy task does the job. - Early recomputation: refresh a value probabilistically before it expires, so a single unlucky request pays the cost while the old value is still serving everyone else.
- Jittered TTLs: add a random spread to expiry so a batch of keys populated together doesn't all expire in the same instant.
I default to coalescing because it's the cheapest to reason about and it also protects you when a downstream dependency is slow rather than merely popular. Jitter is nearly free and I add it almost reflexively. Early recomputation is elegant but adds moving parts, so I only reach for it on the two or three keys that genuinely deserve the attention.
When TTLs aren't enough, invalidate on events
Sometimes half a minute of staleness is genuinely unacceptable. A fraud rule that just got switched off needs to be off now, not after the current TTL burns down. This is where event-driven invalidation earns its keep, and it's also where the nightmares I mentioned actually live.
The pattern is straightforward on paper: a mutation publishes an event, subscribers evict the relevant keys from their local or shared caches. The failure modes are not on paper. Events get dropped. A consumer is deploying and misses the message. The event arrives out of order relative to the database write. Every one of these leaves you stale, and now you've got no TTL backstop because you set a long expiry precisely because you were relying on events.
So my hard rule for event-driven invalidation is that it is an optimization layered on top of a TTL, never a replacement for one. Events make you fresh fast in the common case. The TTL guarantees you converge to correct even when the event machinery fails, which it will. If your correctness depends on a message bus delivering every message exactly once and in order, you have not built a cache, you've built a distributed liability with a latency budget.
Cache keys are an API, so design them like one
The scariest caching bug I've ever shipped was not a stale value. It was a key collision. Two features computed keys that happened to overlap for a subset of inputs, and one feature started serving another feature's data. In a payments context that's not a glitch, that's a potential data-leak incident, and it took us the better part of a day to even believe it was happening.
Treat your key schema as a contract. Namespace everything by domain and version, encode the tenant or merchant boundary explicitly, and bump a version prefix whenever the shape of the cached value changes so an old deploy's entries can't be misread by new code. Something like v3:balance:tenant:{tenantId}:merchant:{merchantId} is verbose and I do not care, because verbose keys have never once cost me a 3 a.m. page and clever short keys have. When you deploy a change to how a value is computed, the version bump is a free, atomic, whole-cache invalidation: old keys are simply never read again and age out on their own.
Know what does not belong in a cache
A cache is for data that is expensive to compute and tolerant of being slightly wrong for a bounded time. That second half is a filter people forget. Some things fail that test hard, and putting them behind a cache is how you generate incidents that look impossible from the logs.
I don't cache authorization decisions with a meaningful TTL. If someone's access is revoked, I do not want a cached "yes" answering for the next ten minutes, so those get either no cache or a cache measured in low single-digit seconds with aggressive event eviction. I don't cache anything I'm about to make a money-moving decision on without reading through to the source of truth first. And I'm wary of caching aggregates that are cheap to get wrong and hard to notice are wrong, like a running total, because those drift silently and nobody spots it until reconciliation. Speed is not worth being confidently incorrect about a balance.
If you can't see staleness, you don't have a cache, you have a mystery
Every cache I run in production emits three numbers at minimum: hit rate, miss rate, and eviction rate. But the one that actually catches invalidation bugs is harder to get and worth the effort: a periodic sampled comparison of a cached value against a fresh read from the source of truth, logging any disagreement that outlives the expected TTL window. A quiet background job that reads a random 0.1% of keys both ways and screams when they differ has caught more real bugs for me than any amount of unit testing.
That merchant balance dashboard I inherited had none of this. It had a hit rate metric and nothing else, which meant it looked perfectly healthy while it was lying. The day we added drift detection, we found three separate keys that had been intermittently stale for months. Nobody had reported two of them. That is the terrifying part of a cache bug: it degrades trust silently, and you only hear about the fraction of cases where someone happens to be looking closely.
Fail closed, or explain yourself later
If I had to compress all of this into one instruction for a team, it's this: make your cache fail closed. Assume every clever invalidation you write will eventually not fire, and design so that when it doesn't, the worst outcome is a value that's a few seconds behind rather than a value that's wrong forever. A short TTL under everything, delete-don't-update on writes, versioned keys, and a drift detector watching your back. None of that is glamorous, and none of it will win a conference talk. But the alternative isn't a slightly slower dashboard. The alternative is a merchant on the phone asking where their 12,000 euros went, and you slowly realizing the money was never missing, your cache just said it was. I've had that call. Build the boring version so you never have to make it.