Caching, replicas & shards: scaling the read and write paths

Follow one request through a system and you get a path: the hops it touches, in order. Most systems have two paths that matter, and they want opposite things.

READ:   client → CDN → app → cache ──miss──→ database replica
WRITE:  client → app → database primary → delete the cached copy
                              └──→ queue → workers (the slow follow-up work)

• The read path wants copies close to the reader. Every copy — a CDN edge, a cache entry, a read replica — takes load off the source of truth and latency off the request. The price is staleness: a copy can be older than the original.
• The write path wants one authoritative place to land, durably. Copying a write doesn't make it faster. You can only make it smaller (do less before replying) or spread it (partition the data so different writes land on different machines).

So the tools sort themselves by path:

The first number to get in any design is the read:write ratio, because it says which half of this table you're shopping in. The URL shortener, at 100 reads per write, lives almost entirely in the top half.

The pattern itself is the NestJS track's Caching & rate limits with Redis unit: read the cache; on a miss, load from the database and leave a copy with a TTL; on a write, update the database first, then delete the key (deletes, unlike sets, can't land in the wrong order). This tier assumes that and asks the design questions around it: what to cache, how big, and what the database behind it has to survive.

What to cache is an estimate too. The shortener stores 6 × 10⁹ links — about 3 TB — but the redirect only needs the links people are clicking now:

Assume: 10 million distinct links are clicked on a given day     (assumption)
Cache:  10,000,000 × ~500 B per entry                            ≈ 5 GB

Five gigabytes fits in one cache node's memory; 3 TB doesn't fit in any. So the cache holds the working set, the TTL decides how long a link stays in it once its clicks stop, and the database still holds everything. Get the assumption from real traffic when you have it.

The hit ratio is the database's load, read upside down. At 10,000 reads a second:

hit ratio 99%  →  1% miss  →    100 reads/s reach the database
hit ratio 98%  →  2% miss  →    200 reads/s   (one point lower, twice the database load)
hit ratio 90%  → 10% miss  →  1,000 reads/s

A dashboard showing 98% looks almost the same as 99%. The database sees its load double. Near the top, read a hit ratio as its complement.

Size the database for a bad day's miss rate, not today's. Three ordinary events push it up:

• a deploy restarts every server, and every in-process cache starts empty;
• a change to key names (a new prefix, a new field in the key) makes every old entry unreachable at once;
• traffic shifts to keys nobody has asked for lately — a newsletter links to old posts.

And the worst day is the cache being gone altogether: 100% misses. The complex tier of A framework for any design does that arithmetic, and AWS's caching article gives the test: run load tests with the cache disabled.

The TTL is a requirement, not a tuning knob. Azure's Cache-Aside guidance is to fit the expiry to how the data is used: too short and requests keep going back to the database, too long and stale data lingers. Delete-on-write only covers writers that go through your code. If another service, an admin script or a migration can change the table, the TTL is the only freshness guarantee those writes get, so pick it by asking "how stale may this be?" — and write the answer down next to the requirement it serves.

Updating the database before deleting the key closes one race (the NestJS track's Caching & rate limits with Redis unit). Two remain, and at fleet scale the fixes have to live in the shared cache, not in one process.

The stale set. A reader that misses can be slower than a writer:

reader A:  GET link:x        → miss
reader A:  SELECT … → old row
                                writer B:  UPDATE … → COMMIT
                                writer B:  DEL link:x
reader A:  SET link:x = old row     ← lands after the delete

The delete happened — and then A refilled the cache with the row it read before the update. Every reader gets the old value until the TTL expires. Facebook's memcache paper calls this a stale set and fixes it with leases: on a miss, the cache hands the reader a 64-bit token bound to the key; a delete invalidates outstanding tokens; a set carrying an invalidated token isn't stored.

Redis has no leases, and a plain version check can't stand in for one: after B's DEL there's nothing left in the cache to compare A's old version against. You can build the lease:

• On a miss, the reader takes a token key: SET lease:link:x <random> NX PX 5000.
• The writer's invalidation deletes lease:link:x along with link:x.
• The fill is a small Lua script that writes link:x only if lease:link:x still holds the reader's token. Redis runs a script atomically, so nothing slips in between the check and the write.

(Versions work if the writer leaves its version behind instead of deleting — the new value, or a tombstone holding the version — and every fill refuses to overwrite a newer version: the version-column idea from Transactions & isolation, applied to the cache.) With neither, the TTL is your only bound on how long a stale set survives, so choose it knowing that.

The stampede across the fleet. A hot key expires and every request that misses asks the database for the same row. In-process coalescing — one load per key per process, like cache-manager's wrap — caps that at one query per server: with 50 servers, a cold key still costs up to 50 identical queries, every time it expires. Removing the multiplier needs a decision every server can see, which means the cache. AWS's Builders' Library describes coalescing done by the servers or by an external cache. Memcache does it with the same leases: a server hands out a token for a key at most once every 10 seconds, and other clients that miss are told to wait briefly. The paper reports a set of keys prone to herds going from a peak of 17K database queries a second without leases to 1.3K with them. The Redis lease above has the same property — whoever loses the NX waits briefly and reads again.