Feature Flags

A feature flag (feature toggle) is a runtime condition in code that turns behavior on or off without deploying a new version. The code for a feature is deployed in a disabled state and enabled later by changing flag configuration. This separates deployment (code reaches production) from release (users see the behavior). In microservices, flags let each service ship continuously while features that span services are enabled together.

The problem flags solve

• Long-lived feature branches delay integration and cause merge conflicts. With flags, incomplete work merges to the mainline behind a disabled flag (trunk-based development).
• A defect found after release requires a rollback or a hotfix deploy. With a flag, the behavior is disabled in seconds, with no deploy.
• A canary deployment selects traffic by instance. A flag selects behavior by user, tenant, or other attributes, at any granularity.

Flag types

Release and experiment flags are temporary. Ops and permission flags are permanent and are managed as configuration.

Rollout mechanics

A flag is evaluated with a context: attributes of the request such as user ID, tenant, region, or application version. Rules are checked in order, and the first match decides the value.

A flag definition lists ordered rules and a default, for example:

{
  "key": "new-checkout",
  "rules": [
    { "if": { "user_id": ["u-1", "u-2"] }, "value": true },
    { "if": { "country": ["DE"] }, "value": false },
    { "percentage": 10, "value": true }
  ],
  "default": false
}

A request from user u-1 matches the first rule and gets true. A German user not on the allowlist matches the second rule and gets false. Other users fall into the percentage rule, and users outside the 10% receive the default.

Common targeting rules:

• Enabled for internal users or a specific allowlist.
• Enabled for a segment, such as one tenant or country.
• Percentage rollout: enabled for a fixed share of users.

Deterministic bucketing

A percentage rollout must give each user the same answer on every request and every server. Random selection per request would make the feature appear and vanish for the same user. Instead the evaluator hashes a stable identifier together with the flag name and maps the hash to a bucket from 0 to 99. A user is included if the bucket is below the rollout percentage.

Including the flag name in the hash makes bucket assignments independent across flags. Otherwise the same users would always be first in every rollout. Raising the percentage only adds users, because a user in bucket 12 stays included whenever the percentage is above 12.

import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.stream.IntStream;

public class Main {
    static long fnv1a(byte[] data) {
        long hash = 0xcbf29ce484222325L;
        for (byte b : data) {
            hash ^= (b & 0xffL);
            hash *= 0x100000001b3L;
        }
        return hash;
    }

    static int bucket(String flag, String userId) {
        long hash = fnv1a((flag + ":" + userId).getBytes(StandardCharsets.UTF_8));
        return (int) Long.remainderUnsigned(hash, 100);
    }

    static boolean enabled(String flag, String userId, int percent) {
        return bucket(flag, userId) < percent;
    }

    public static void main(String[] args) {
        List<String> users = IntStream.range(0, 1000).mapToObj(i -> "user-" + i).toList();
        for (int percent : new int[] {10, 50}) {
            long on = users.stream().filter(u -> enabled("new-checkout", u, percent)).count();
            System.out.println(percent + "% rollout -> " + on + " of 1000 users enabled");
        }
        // Stable: same input gives the same answer every time.
        System.out.println("stable: "
                + (enabled("new-checkout", "user-7", 50) == enabled("new-checkout", "user-7", 50)));
        // Monotonic: everyone enabled at 10% stays enabled at 50%.
        boolean monotonic = users.stream()
                .allMatch(u -> !enabled("new-checkout", u, 10) || enabled("new-checkout", u, 50));
        System.out.println("monotonic: " + monotonic);
    }
}

10% rollout -> 90 of 1000 users enabled
50% rollout -> 482 of 1000 users enabled
stable: true
monotonic: true

The hash function must be stable across languages, versions, and processes. The default Object.hashCode() is identity-based and differs between runs, so it is unsuitable. The example uses 64-bit FNV-1a, a fixed algorithm that needs only the JDK; Long.remainderUnsigned keeps the bucket non-negative, since the Java long is signed. Production SDKs use fixed algorithms such as MurmurHash3 or SHA-1.

Kill switches and dark launches

A kill switch is an ops flag that disables a feature or a dependency call immediately. It is defined ahead of time for expensive or risky paths, and it usually degrades gracefully, for example by returning cached data or hiding a recommendation panel (see bulkhead, fallback, and load shedding). A kill switch must have a safe default that applies when the flag service is unreachable.

A dark launch deploys a feature and exercises it with production traffic without exposing results to users. For example, a new pricing service is called on each request and its answer is compared to the old service's answer and logged, while users still receive the old answer. This validates correctness and performance under real load. Like shadow traffic, it must avoid duplicating side effects.

A typical staged release: internal users, then 1%, 10%, 50%, 100%, with metrics compared at each stage. If error rate or latency regresses, set the flag to off.

Evaluation architecture

graph LR
    UI["Flag management UI / API"] --> ST[("Flag store")]
    ST -->|"stream or poll"| SDK1["Service A: SDK with local cache"]
    ST -->|"stream or poll"| SDK2["Service B: SDK with local cache"]
    SDK1 --> E1["Evaluate rules in process"]
    SDK2 --> E2["Evaluate rules in process"]

Requirements for a reliable design:

• In-memory cache with a default. If the flag store is unavailable, the SDK serves the last known rules, or the coded default.
• No blocking network call on the request path for local evaluation.
• Consistent evaluation within a request. Read each flag once per request and pass the value along, so a mid-request flag change does not cause inconsistent behavior.
• Context propagation. When a feature spans services, evaluate once at the edge and pass the result in a request header, or make every service evaluate the same rule on the same user ID, so that all services agree.
• Audit log and access control. Flag changes affect production behavior, so record who changed what and when.
• Metrics per flag value. Tag metrics and traces with the flag variant to compare variants.

Flag debt

Every flag adds a branch, doubling the number of code paths tested for that flag. Flags that outlive their purpose become flag debt: dead branches, unclear behavior, and risk when someone flips an old flag.

Controls:

• Record an owner, a creation date, and an expected removal date for every temporary flag.
• Remove the flag and the losing code path once the rollout reaches 100% and is stable.
• Alert on flags past their expiry.
• Limit interactions between flags. Combinations grow exponentially, and only a few are tested.
• Keep flag logic at few points in the code, for example at a decision boundary, not scattered through business logic.
• Test both flag states in CI.

When not to use flags

• For long-lived configuration values that are not toggles, use configuration management.
• As a substitute for compatible API and schema changes. A flag does not remove the need for backward-compatible database migrations.
• For security enforcement. Authorization belongs in an authorization system, not in a toggle.

Interview questions

• How do you give a user a consistent experience during a percentage rollout? Hash a stable user identifier with the flag name into a fixed bucket and compare it with the rollout percentage. The same user always gets the same result, and increasing the percentage only adds users.
• What should happen if the flag service goes down? Services keep evaluating with cached rules, or fall back to coded defaults chosen to be safe. Flag evaluation must not fail requests.
• How do flags differ from canary deployments? A canary shifts traffic between versions of the code. A flag switches behavior within one version, with targeting by user or attribute, and can be changed without a deploy.
• Why remove old flags? Each flag adds untested code paths and confusing behavior. Removing finished flags keeps the code paths and risks limited.