How We Run Blue-Green Deployments

The first time I ran a blue-green deployment in anger, it was 6:40 on a Friday evening and the payment authorization service had been down for eleven minutes. We had shipped a schema change that a rollback script couldn't reverse cleanly, and half our merchants were seeing declined transactions on perfectly good cards. That night cost us a chargeback investigation, a very unhappy call with an acquiring bank, and a weekend I'd rather forget. It also convinced me that the deployment strategy we'd inherited was not fit for money.

Blue-green is not a silver bullet, and anyone who sells it as one hasn't run it under real load. But used with discipline, it turns a deployment from a held-breath event into something closer to boring. In regulated fintech, boring is the highest compliment I can pay to a release process.

What blue-green actually means to us

The textbook version is simple: you keep two identical production environments, call them blue and green. One serves live traffic while the other sits idle. You deploy the new version to the idle one, verify it, then flip a router so all traffic moves to the freshly deployed environment. The old one stays warm for a while in case you need to flip back.

Our version is a little less pure. We run on AWS behind an Application Load Balancer with two target groups, and the "flip" is a weighted change to the listener rules. We rarely go from 0 to 100 in one move, because moving every session at once is how you find out your connection pools weren't sized for a cold start. What matters is the core property: the new version is fully running and validated before it takes a single real request, and reverting is a routing change, not a rebuild.

Why not just roll or canary

I get asked this constantly, usually by engineers who like canary deploys and think blue-green is heavyweight. They're not wrong that it costs more. You are, by definition, paying for roughly double the compute during the overlap window. For a team counting every dollar of cloud spend, that's a real conversation.

But here's where I land, and I'll be blunt about it: for the services that touch money, I want the ability to revert in under thirty seconds without redeploying anything. Rolling deployments smear the change across your fleet, so a bad release means some pods are poisoned and some aren't, and your rollback is another slow roll. Canary is genuinely excellent for stateless read-heavy APIs where you can watch error rates climb on 5% of traffic. It's much less pleasant when the failure mode is a subtle double-charge that doesn't show up as an HTTP 500. Blue-green gives me one clean line: this whole environment is either good or it isn't.

The database is the whole problem

Everything easy about blue-green is the stateless part. Everything hard is the database, because you cannot keep two live copies of your ledger and flip between them. Both environments talk to the same database, which means your schema has to be compatible with both the old and new application versions at the same time.

This forces a discipline I've grown to love: expand and contract. You never rename or drop a column in the same release that stops using it. You add the new thing, deploy code that writes to both and reads from the new one, and only in a later release do you remove the old column. It's slower. It also means a rollback never leaves the database in a state the previous code can't understand.

The rule I hammer on with every new engineer: a schema migration and the code that depends on it must never ship in the same deploy. If you can't roll back the code without a database change, you don't have blue-green. You have a magic trick that works until it doesn't.

Here's the shape of an expand migration we'd actually run. Note that it's additive and safe against the currently-live code, which knows nothing about the new column:

-- Release N: expand only. Old code still runs happily.
ALTER TABLE settlements
    ADD COLUMN settlement_reference VARCHAR(64) NULL;

CREATE INDEX CONCURRENTLY ix_settlements_reference
    ON settlements (settlement_reference)
    WHERE settlement_reference IS NOT NULL;

-- New code backfills and dual-writes.
-- Release N+2, after the flip has been stable for days: contract.
-- ALTER TABLE settlements DROP COLUMN legacy_ref;  -- only now

The cutover, step by step

The actual flip is the least interesting part when you've done the preparation, and that's the goal. Our runbook is deliberately mechanical. When someone is moving production traffic between environments at 2 in the afternoon, I do not want them making judgment calls under pressure.

• Deploy the new build to the idle (green) environment and wait for health checks to pass on every instance.
• Run the smoke suite against green's internal hostname. This includes a real tokenized test authorization against the sandbox acquirer, not just a ping.
• Shift 10% of traffic. Watch error rate, p99 latency, and authorization success rate for five minutes. We alert if auth success drops more than 0.3% from baseline.
• Shift to 50%, hold for five minutes, then 100%.
• Leave blue running, drained but warm, for one hour minimum. On big releases, until the next morning.

The step people want to skip is the warm hold on the old environment. Don't. The whole point of paying for two environments is that reverting is instant, and that property evaporates the moment you tear down blue to save money. I've watched a team scale their old environment to zero eight minutes after cutover, hit a problem at minute twelve, and then spend twenty minutes cold-starting the thing they could have kept alive for the price of a few idle instances.

What flipping back actually costs

People talk about rollback like it's free. It isn't, and pretending otherwise is how you end up flip-flopping traffic three times in an hour and confusing everyone including yourself. A flip back is cheap technically but expensive in trust. Every reversal is a signal to the on-call engineer and to the wider team that the release process let something through.

So we treat a flip back as a real incident, even when it takes eight seconds. It gets a ticket, a short writeup, and a look at why the smoke suite didn't catch whatever we caught in production. About a third of our reversals over the last year traced back to something that was genuinely unobservable in a non-production environment, usually a data-shape assumption that only real merchant traffic exposes. The other two thirds were things we could have caught, and those are the ones worth being annoyed about.

Sessions, background jobs, and in-flight work

The load balancer flip handles new requests cleanly. It does nothing for the work already running. If you have a job that started on blue and takes ninety seconds, and you flip to green at second forty, that job needs to finish on blue or you'll have a half-processed batch. Connection draining on the target group handles most of it, but you have to actually configure the deregistration delay long enough to cover your longest in-flight request. Ours is set to 300 seconds, which is generous, but a stuck settlement batch is not where I want to economize.

Background workers are their own headache because they don't sit behind the load balancer at all. We run them as a separate deployment with its own blue-green flip, and critically, we make every job idempotent with a natural key so that a message processed by both old and new workers during the overlap doesn't double-book. If your workers aren't idempotent, blue-green will find that out for you, at the worst possible time.

The observability you need first

You cannot run this safely without a baseline to compare against. Before we ever shift traffic, we're staring at a dashboard that shows the same four metrics for blue and green side by side: error rate, p99 latency, authorization success rate, and requests per second. The comparison is the whole game. A 2% error rate on green means nothing until you see that blue is also sitting at 2% because a downstream provider is having a rough afternoon.

Here's the small piece of instrumentation that pays for itself every single deploy. We tag every request with the environment that served it, so we can slice any metric by color:

public class EnvironmentTagMiddleware
{
    private readonly RequestDelegate _next;
    private readonly string _color;

    public EnvironmentTagMiddleware(RequestDelegate next, IConfiguration config)
    {
        _next = next;
        _color = config["DEPLOY_COLOR"] ?? "unknown";
    }

    public async Task InvokeAsync(HttpContext ctx, IMetrics metrics)
    {
        using (metrics.Measure.Timer.Time("request_duration",
            new MetricTags("color", _color)))
        {
            ctx.Response.Headers["X-Deploy-Color"] = _color;
            await _next(ctx);
        }
    }
}

That response header alone has saved us hours. When a merchant reports a weird error, the first thing support asks for is the X-Deploy-Color value, and half the time it tells us exactly which environment and therefore which release is involved.

Where I don't use it

I'd be lying if I said we run blue-green everywhere. We don't, and I'd push back on anyone who wanted to. For internal tooling, the reporting dashboard, the batch reconciliation jobs that run overnight with no user waiting, a plain rolling deploy is fine and the doubled cost isn't justified. Reserve the heavy machinery for the paths where a bad minute has a dollar value attached.

There's also a scale below which it's overkill. If you're a five-person team with one service and ten thousand transactions a day, the operational overhead of maintaining two environments and expand-contract migrations will slow you down more than it protects you. I'd rather that team ship a good rollback script and a maintenance window and revisit the question when the numbers get scary. The strategy has to match the stakes.

Conclusion

The thing nobody tells you is that blue-green deployments are mostly a forcing function for other good habits. The flip itself is trivial. What makes it work is backward-compatible migrations, idempotent workers, honest observability, and the discipline to keep the old environment warm when you're itching to reclaim the compute. If you adopt the router trick without those, you've bought yourself a false sense of safety at double the price. Get the habits first, and the two environments become almost incidental. That's the part I wish someone had told me before that Friday night.