Feature Flags Done Right

Every engineering team I have led eventually reaches the same uncomfortable moment: a release is ready, the code is merged, and yet nobody is quite willing to ship it. The change is too large, the blast radius too uncertain, the rollback story too painful. In a regulated payments business, that hesitation is rational. The cost of a bad deploy is not a few angry tweets; it is settlement delays, reconciliation breaks, and a compliance conversation you would rather not have.

Feature flags are the tool that turns that hesitation into confidence. Used well, they decouple deployment from release, let you test in production safely, and give you an instant kill switch when something goes wrong. Used badly, they become a second, undocumented configuration system that nobody trusts and nobody cleans up. This is how I think about getting them right.

Deployment Is Not Release

The single most valuable idea behind feature flags is that deploying code and releasing a feature are two separate events. When you merge a payment-routing change behind a flag that defaults to off, you can deploy it to production at 10am with the rest of your continuous integration pipeline, and then turn it on for one internal merchant at 3pm after your risk team has reviewed the dashboards. Nothing about that second step requires a build, a deploy window, or an on-call engineer staring at logs at midnight.

This separation changes the emotional temperature of shipping. A deploy that contains twelve merged pull requests, all dark, is far less frightening than a deploy that flips twelve behaviours at once. The code path exists in production, exercised by your health checks and integration suites, long before any real customer money flows through it. By the time you flip the flag, you have already removed most of the unknowns that make releases scary.

The goal of a flag is to make the act of releasing boring. If turning a feature on still feels dramatic, you have not finished de-risking it.

Four Kinds Of Flags

One reason flag systems rot is that teams treat every flag identically when they are, in fact, very different animals with very different lifespans. I insist that every flag declare its type at creation time, because the type tells you who owns it and when it should die.

• Release flags wrap a new feature during rollout. They are short-lived and should be removed once the feature is fully on and stable, usually within a sprint or two.
• Operational flags are kill switches and circuit breakers for systems you want to disable under load or during an incident. They live indefinitely and are owned by the platform team.
• Permission flags gate access by plan, merchant tier, or regulatory jurisdiction. They are effectively long-term product configuration and rarely get removed.
• Experiment flags drive A/B tests and carry a built-in expiry tied to the experiment window.

When someone proposes a new flag, the first question is which of these it is. A release flag that is still in the codebase six months later is a defect; a permission flag that lives that long is doing its job. Conflating the two is how you end up with a flag named temp_new_checkout that has been load-bearing for two years and that nobody dares delete.

Flags As Code, Not Mystery Config

A flag evaluation should look like ordinary, readable code at the call site, and the surrounding logic should still make sense if the flag were deleted. I am wary of clever wrappers that hide which branch is the new one. The point of a flag is clarity under pressure, and at 2am during an incident you want the code to be obvious.

Here is the shape I prefer in our .NET services. The evaluation is explicit, the context carries enough to make targeting decisions, and there is a sane default if the provider is unreachable.

public async Task<PaymentResult> RoutePaymentAsync(PaymentRequest request)
{
    var context = new FlagContext
    {
        MerchantId = request.MerchantId,
        Country = request.Country,
        Amount = request.Amount
    };

    // Default to the proven path if evaluation fails.
    bool useNewRouter = await _flags.IsEnabledAsync(
        "release.smart-routing-v2", context, defaultValue: false);

    if (useNewRouter && request.Amount <= _config.SmartRoutingCeiling)
    {
        return await _smartRouter.RouteAsync(request);
    }

    return await _legacyRouter.RouteAsync(request);
}

Notice that the default value is passed in explicitly at the call site. If the flag service times out, this transaction quietly takes the legacy path rather than throwing. In a payments system, a flag provider outage must never become a payments outage. The dependency on the flag system has to fail safe, and the only way to guarantee that is to decide the fallback at every call site rather than hoping the SDK does the right thing.

Targeting And Progressive Rollout

A boolean on/off flag is the least interesting thing you can do. The real power comes from targeting: turning a feature on for a specific merchant, a percentage of traffic, a country, or a cohort that matches some rule. This is what lets you do a genuine progressive rollout rather than a binary flip.

My standard playbook for a risky change is to ramp deliberately. Internal test merchants first, then a single friendly design-partner merchant who knows they are early, then one percent of traffic, then five, twenty-five, fifty, and finally everyone. At each step I am watching the same dashboards: error rate, latency, authorisation rate, and any reconciliation discrepancy. The percentage is not the point; the observation between steps is the point. If you ramp from one to one hundred percent in ten minutes, you have a fancy on/off switch, not a rollout.

Crucially, the targeting must be sticky. If a merchant is in the treatment group, they should stay in it across requests and across server restarts, which means bucketing on a stable hash of the merchant identifier rather than a random draw per call. A customer who sees the new checkout on one request and the old one on the next will file a confused support ticket, and in a financial product that inconsistency erodes trust faster than a slightly slower page.

The Cost Of Forgotten Flags

Every flag is a branch, and every branch multiplies the number of states your system can be in. Ten independent boolean flags describe over a thousand possible configurations, and you are almost certainly not testing all of them. Most are nonsensical, but the combinatorial explosion is real, and it shows up as bugs that only reproduce for a specific merchant whose flag combination nobody anticipated.

This is why flag debt is technical debt with a particularly nasty interest rate. A stale release flag does not just clutter the code; it doubles the test matrix for everything it touches, it confuses new engineers about which path is canonical, and it creates a tempting but dangerous lever that someone will eventually pull during an incident without understanding the consequences. I have seen a long-forgotten flag, flipped in a panic, take down a service it had nothing to do with because the dead code path it guarded had silently bit-rotted.

A flag with no expiry date and no owner is not a feature flag. It is a hidden configuration option masquerading as one, and it will outlive everyone who understood it.

Lifecycle And Cleanup

Because flags rot, the discipline that matters most is not how you create them but how you retire them. I treat flag cleanup as a first-class part of the work, not an afterthought. When an engineer opens a pull request introducing a release flag, our convention is that the same ticket carries a follow-up subtask to remove it, and that subtask has a due date. A flag without a removal plan does not pass review.

We also run automated hygiene. A weekly job queries our flag provider for flags that have been at one hundred percent for more than thirty days and posts the list to the owning team's channel. The query against our flag audit table is straightforward and worth surfacing, because making staleness visible is most of the battle.

SELECT f.flag_key,
       f.owner_team,
       f.created_at,
       f.last_changed_at
FROM   feature_flags f
WHERE  f.rollout_percentage = 100
  AND  f.flag_type = 'release'
  AND  f.last_changed_at < NOW() - INTERVAL '30 days'
ORDER  BY f.created_at ASC;

The cultural rule behind the automation is simple: removing a flag is a celebration, not a chore. When a release flag comes out and the old code path is deleted, that pull request is the moment the feature truly ships. I make a point of recognising that work in the same way I recognise the feature launch itself, because otherwise cleanup is the thing that always loses to the next deadline.

Auditability In Regulated Environments

In a fintech, a feature flag change can be a material change to how money moves, and that means it falls squarely within the same governance expectations as a code deploy. If we change the authorisation logic for a class of transactions by flipping a flag, an auditor will reasonably ask who made that change, when, on what authority, and how it was reviewed. "Someone toggled it in the UI" is not an acceptable answer.

So every flag change in our production environment is recorded with the actor, the timestamp, the previous and new values, and a reason field that is mandatory rather than optional. High-impact flags, the ones touching settlement, fraud, or routing, require a second approver before they can change in production, exactly like a code review. The flag system is wired into the same audit log as the rest of our infrastructure, so a regulator or an internal compliance review can reconstruct the complete history of how a given behaviour was governed over time.

This sounds heavy, and for the operational kill switches it would be, so we tier it. A low-risk experiment flag can be changed freely by the product team. A flag that alters how we calculate fees or where funds settle gets the full four-eyes treatment. Matching the control to the risk is what keeps the governance credible rather than ceremonial; if you make everything require two approvers, people route around the system, and then you have no audit trail at all.

Testing With Flags Present

A flag that is never tested in both states is a liability, because the off state will quietly diverge from reality while everyone's attention is on the new path. My rule is that any flag controlling meaningful behaviour must have automated tests covering both states for as long as both branches exist in the code. The moment the old branch is deleted, those tests go too, which is another reason to keep flag lifespans short.

For release flags mid-rollout, our continuous integration runs the relevant suites twice, once with the flag forced on and once forced off, so a regression in either path fails the build. This catches the classic mistake where the new path works beautifully but the developer accidentally broke the fallback they assumed they would never need. In a payments system that fallback is precisely the path you will rely on during an outage, so it deserves more testing scrutiny, not less.

Conclusion

Feature flags are not a silver bullet, and they are emphatically not free. Every flag you add is a branch in your logic, a row in your audit log, and a small ongoing tax on comprehension. The teams that get value from them are the ones that treat that cost seriously: they classify flags by type, they default to the safe path when the flag system is unavailable, they roll out progressively while actually watching the numbers, they govern high-impact changes with the same rigour as code, and above all they delete flags as soon as the job is done. Get that discipline right and flags become what they should be: the mechanism that lets a careful, regulated engineering organisation ship confidently and often, while keeping a steady hand on the kill switch. That, more than any clever targeting rule, is what feature flags done right actually buys you.