The first time I took down a payments platform during business hours, it was for a version bump. Not a migration, not a big-bang rewrite, just a routine deploy that everyone assumed was safe. We drained one node, brought it back on the new build, and discovered the new build wrote a settlement record in a shape the old nodes couldn't read. For about nine minutes we had two versions of the truth in the same cluster, and reconciliation spent the next three days sorting it out.
That outage taught me more about deploys than any book did. Zero-downtime is not a feature you turn on. It is a discipline that touches your database, your load balancer, your feature flags, and mostly your own patience. The turning-it-on part takes an afternoon. The discipline takes years. Here is how I actually do it, and where I have watched it go wrong.
What Zero-Downtime Actually Means
People say "zero-downtime" and mean wildly different things. Some mean the health check never flaps. Some mean no user ever sees a 500. In regulated fintech I hold a stricter line: no in-flight request is dropped, no transaction is lost or duplicated, and no client sees an inconsistent view of their money for even a second. A dropped API call to a marketing site is a shrug. A dropped call to an authorization endpoint is a chargeback, a support ticket, and sometimes a regulator asking questions.
So when I audit a deploy process, the question I ask is not "does the site stay up." It is "if we deploy at the exact moment a customer submits a large transfer, does that transfer complete exactly once with the correct outcome." If the honest answer is "probably," you don't have zero-downtime deploys. You have lucky deploys.
The Load Balancer Is Not Enough
The naive picture of zero-downtime is a load balancer and a rolling restart. Spin up new instances, wait for them to pass health checks, drain the old ones, done. This works right up until it doesn't, and the gap is almost always connection draining. Your load balancer stops sending new requests to the old node, but that node still has forty requests in flight, three of them long-running report generations that take twenty seconds.
If your shutdown handler kills the process the moment it gets SIGTERM, those forty requests die. In .NET this is exactly what IHostApplicationLifetime and graceful shutdown timeouts exist to solve, and yet I keep finding services where nobody set them. The default host shutdown timeout is five seconds. If your slowest request is longer than that, you are cutting requests off mid-flight on every single deploy and calling it zero-downtime.
// Program.cs — give in-flight work time to finish
builder.Host.ConfigureHostOptions(options =>
{
// Match this to your slowest realistic request,
// plus headroom. We use 45s for report endpoints.
options.ShutdownTimeout = TimeSpan.FromSeconds(45);
});
// Stop advertising healthy BEFORE the drain begins,
// so the LB pulls this node while requests finish.
app.Lifetime.ApplicationStopping.Register(() =>
{
HealthState.SetNotReady(); // readiness probe now fails
Thread.Sleep(TimeSpan.FromSeconds(5)); // let LB notice
});The Real Enemy Is the Database
Stateless app servers are the easy part. You can run ten versions of a stateless service side by side and nobody cares. The database is where zero-downtime deploys go to die, because there is exactly one of it and every version of your code has an opinion about its schema.
The rule I drill into every team I lead: your schema and your code must be compatible in both directions during a deploy. For the entire duration of a rolling deploy, the old code and the new code are both live, both hitting the same tables. If the new code renames a column, the old code breaks. If the new code adds a NOT NULL column with no default, the old code's inserts fail. This is the settlement-record bug I opened with, dressed in different clothes.
Never deploy a schema change and the code that depends on it in the same release. The schema goes first, backward-compatible. The code that requires the new shape goes second, once every node is already tolerant of it.
Expand and Contract
The pattern that makes database changes safe is expand-and-contract, sometimes called parallel change. You never mutate a column in place. You expand the schema to support both the old and new world, migrate the data, shift the code over, and only then contract away the old shape. It is slower and it feels like bureaucracy the first few times. It is also the difference between a boring Tuesday and a reconciliation nightmare.
Say you want to split a single full_name column into first_name and last_name. The tempting move is one migration that does it all. The safe sequence is more like this:
- Add the two new nullable columns. Deploy. Nothing reads them yet.
- Deploy code that writes to both the old and new columns on every update (dual-write).
- Backfill existing rows in batches, off-peak, throttled so you don't saturate the write-ahead log.
- Deploy code that reads from the new columns but still writes both.
- Once you are confident, stop writing the old column and, in a much later release, drop it.
Five deploys to move one column. Yes, five. And every one of them is reversible, and none of them takes the system down. I have shipped column splits on a table doing two thousand writes a second this way without a single failed request. The backfill is the part people rush; batch it, add a sleep between batches, and watch replication lag like a hawk.
Long Migrations Will Lock You Out
Even a backward-compatible migration can cause an outage if it holds the wrong lock. This is the sharp edge people don't see coming. Adding an index on a large table in PostgreSQL without CONCURRENTLY takes an exclusive lock and blocks writes for as long as the build takes. On a big table that can be minutes, and minutes of blocked writes on a payments table is an outage no matter how healthy your app servers look.
-- Wrong: locks the table against writes for the whole build
CREATE INDEX idx_tx_account ON transactions (account_id);
-- Right: no exclusive lock, other writes proceed
CREATE INDEX CONCURRENTLY idx_tx_account ON transactions (account_id);
-- And always set a short lock timeout so a migration
-- that CAN'T get its lock fails fast instead of queuing
-- behind every query on the table.
SET lock_timeout = '3s';
ALTER TABLE transactions ADD COLUMN reconciled_at timestamptz;The lock_timeout line is the one I have made non-negotiable. Without it, a migration that can't acquire its lock will sit at the front of the queue, and every query behind it stacks up until connections exhaust and the whole database appears to hang. With a three-second timeout, the migration simply fails, your deploy aborts, and you fix it. Fail fast beats fail mysteriously.
Decouple Deploy From Release
The single most useful mental shift I have made is separating deploy from release. Deploying means the code is running on production servers. Releasing means users can reach the new behavior. When you conflate the two, every deploy is a high-stakes event. When you separate them with feature flags, deploys become boring and releases become deliberate.
We ship new payment flows behind a flag that defaults to off. The code deploys during the day, sits there dark, and gets exercised by internal test traffic and a handful of employee accounts. When we turn it on, we turn it on for one percent of merchants, watch the authorization success rate and latency for an hour, then ramp. If something is wrong we flip the flag, not the deploy. A flag flip is instant and touches nothing structural. A rollback deploy is slow and, if a migration already ran, sometimes not even possible.
Rollback Is a Design Constraint, Not a Button
Everyone wants a rollback button. Few teams design for one. The uncomfortable truth is that once a migration has changed data, you often cannot cleanly roll back the code, because the old code doesn't understand the new data. This is why expand-and-contract matters so much: at every intermediate step, both the previous and current code work against the current schema, so rolling the code back is always safe.
I treat rollback as a property I have to preserve deploy by deploy, not a capability I assume exists. Before any release I ask a blunt question: if this is live for twenty minutes and writes real data, can I go back to the previous version without corrupting anything? If the answer is no, the release gets restructured until the answer is yes. That discipline is annoying and it has saved me more times than I can count.
Prove It, Then Watch It
You cannot claim zero-downtime from a diagram. You have to prove it. We run a synthetic client that fires a steady stream of representative transactions during every deploy to staging, and it asserts that not one of them fails or returns an inconsistent result. If a deploy drops even a single request, the pipeline goes red. That test has caught draining bugs, connection-pool resets, and one memorable case where a config reload silently dropped every open gRPC stream.
In production the equivalent is tight monitoring during the deploy window itself. Watch error rate, p99 latency, and connection counts per node as traffic shifts. The thing I care about most is the shape of the cutover: a clean deploy shows a smooth handoff, while a broken one shows a spike of 502s exactly when the old nodes drain. If you can see that spike, your draining is wrong, full stop, no matter what the aggregate dashboard says a minute later.
Conclusion
Zero-downtime deploys are not really about uptime. They are about respect for the fact that at the moment you deploy, a real person is halfway through moving real money, and they did not agree to be your test case. Get the load balancer draining right, treat the database as the constraint it is, decouple deploy from release, and preserve rollback at every step. Do that and deploys stop being events you schedule for 2 a.m. and dread. The best compliment my team ever got was from an auditor who asked how often we deploy and, hearing "several times a day, during business hours," assumed we had misunderstood the question. We hadn't. That is the whole point.