60% of our Postgres reads moved to a replica. Here’s what broke along the way.

We moved most of our reads to a Postgres replica. Then things started disappearing.

Not actually disappearing. But occasionally, one of our workers would try to fetch a record that we knew existed and get a NotFoundError.

The problem was a race we hadn't fully accounted for.

A request would write to the primary → publish an event → a worker would pick up that event → query the replica.

Sometimes our workers were fast enough to process the event before the replica had caught up with the write.

It only happened on ~0.1–0.5% of messages, but that was enough to make moving more traffic to the replica risky.

Our solution was to use Postgres Log Sequence Numbers (LSNs).

Every message we publish gets stamped with the primary's current LSN. When a worker picks it up, we compare that against the replica's LSN.

If the replica has caught up, process the message.

If it hasn't, put the message back on the queue and try again later.

That also gave us a useful backpressure mechanism: if the replica starts falling behind during a period of heavy load, work simply queues until it catches up instead of spilling back onto the primary and potentially making things worse.

We eventually extended the same idea to our API and made the replica the default destination for reads across our codebase.

Today, more than 60% of our read queries go to the replica, and we cut CPU utilization on our primary database roughly in half.

We wrote up the whole journey, including the consistency issues we hit, LSN stamping, connection pool routing, and some surprisingly effective optimizations:

https://incident.io/blog/dont-add-a-read-replica-until-youve-read-this

For those running Postgres at scale: how are you handling read-after-write consistency with replicas?