Feature Flags: Safe Deployments, Dark Launches, and Rolling Rollouts
Every deployment is a risk event. Something that worked in staging breaks in production. A new checkout flow confuses users. A database query that ran fast on 1,000 rows crawls on 10 million. The classic response is to slow down deployments — review more, test more, deploy less often. Feature flags take the opposite approach: deploy constantly, but control what users actually see through configuration rather than code.
This post covers the full lifecycle of feature flags, from a minimal PostgreSQL toggle table to production-grade platforms like Unleash and LaunchDarkly, with working code examples in Python, Go, and TypeScript.
What Are Feature Flags and Why Do They Matter?
A feature flag (also called a feature toggle, feature switch, or feature gate — they all mean the same thing) is a conditional in your code that gates access to a feature based on runtime configuration rather than compiled behavior.
The core idea is simple:
|
|
That conditional is evaluated at runtime against a configuration store. Flip the configuration, and every user hitting that code path gets the new behavior — no deployment required.
The Deployment Risk Problem
The traditional mental model is: deploy = release. Code lands in production, users get it. This creates enormous incentive to batch changes together (“let’s ship these 10 things at once”) and to slow down deployments with heavyweight review processes.
Feature flags break this coupling:
- Deploy means: land code in production. The new code path exists but runs for nobody.
- Release means: turn the flag on. Users start seeing the feature.
These two events can be separated by minutes, hours, or weeks. The code sits dark in production until you decide to light it up.
Use Cases
Kill switch — The highest-value use case. Something is broken in production. With a kill switch, you flip the flag off and the feature is gone in seconds — no incident war room, no emergency rollback, no 3am deploy. The blast radius of any bug is bounded by how quickly you can flip a flag.
Canary release — Enable a feature for 1% of users, watch your error rate and latency dashboards, increase to 5%, 25%, 50%, then 100%. If anything looks wrong, drop it back to 0% immediately. This is how every serious internet company ships.
A/B testing — Show version A to 50% of users and version B to the other 50%. Measure conversion rate, click-through, revenue. Let data drive product decisions instead of opinions.
Dark launch — Run the new code path in production alongside the old one, but only return results from the old path to users. The new code runs silently, its output compared against the old code in the background. You get production load testing and result validation before any user sees the new behavior.
Beta programs — Enable a feature for specific user accounts before general availability. Your design partners, internal employees, and power users get early access; everyone else sees nothing different.
Ops flags — Tune system behavior without a deploy. Rate limits, cache TTLs, connection pool sizes, timeout thresholds — these are operational parameters that have no business being baked into compiled binaries.
The Trunk-Based Development Connection
Feature flags are what make trunk-based development practical at scale. Without them, incomplete features have to live on long-running branches, causing merge conflicts and integration pain. With flags, you merge incomplete work to main every day — the new code ships to production with the flag off, and the flag goes on only when the feature is complete and you’re ready.
Facebook has shipped features to 1% of users first for well over a decade. GitHub used flags extensively to ship GitHub Actions, gradually expanding access from private beta to GA over months — all from the same codebase.
Types of Feature Flags
Not all flags are the same. Mixing these categories in your head leads to flags that never get cleaned up.
Release Flags (Temporary)
Release flags hide incomplete or not-yet-released features during development. They exist to enable continuous integration of partial work. They are always temporary — once the feature is fully released to 100% of users, the flag should be removed along with the old code path.
Typical lifespan: 1-8 weeks. If a release flag is still around after three months, something has gone wrong.
Experiment Flags (Temporary)
Experiment flags drive A/B tests and multivariate experiments. They are temporary because once the experiment concludes and a winner is chosen, the losing variant is deleted and the flag goes away.
Statistical significance matters here. An experiment flag that gets removed after one day of data is not producing useful signal. Most teams need at least a week of data per variant, accounting for day-of-week variation, before drawing conclusions.
Ops Flags (Permanent)
Ops flags control operational behavior — they are your circuit breakers, your emergency throttles, your configuration surface. A flag that sets the maximum number of concurrent requests to an upstream service is an ops flag. It is expected to live forever as part of the system’s operational interface.
These flags should be documented alongside your runbooks. When an incident happens at 2am, the on-call engineer needs to know: “flip this flag to shed load.”
Permission Flags (Permanent)
Permission flags enable features for specific user segments: premium subscribers, internal employees, beta participants, specific enterprise accounts. They are permanent because the segmentation logic is permanent.
The Flag Debt Problem
Here is the danger: flags accumulate. Teams create them, forget to remove them, and six months later nobody knows what’s on or off or why. This is called flag debt, and it is genuinely hazardous.
The most cited example is Knight Capital Group in 2012. A deployment flag controlling which code path the trading system used was accidentally re-enabled on a production server. The system executed orders at high speed using logic that should have been long dead — $440 million in losses in 45 minutes. A leftover deployment flag, improperly managed.
The fix is simple but requires discipline: set a removal date when you create a release or experiment flag. Put it in the flag description. Schedule a calendar reminder. Treat flag removal as part of the feature’s definition of done.
A Simple Toggle Table: Rolling Your Own
For small teams, simple use cases, or environments where you can’t send feature evaluation data to a third-party SaaS, a database table and a few dozen lines of code are often all you need.
PostgreSQL Schema
|
|
The rollout_pct column enables percentage-based rollouts: when set to 10, only 10% of users see the feature. The key trick is making that 10% deterministic — the same user must always land in the same bucket, otherwise users experience flickering behavior.
Python Implementation with Redis Caching
Hitting the database on every flag check is too slow for high-throughput services. Cache flag state in Redis with a short TTL.
|
|
The _user_bucket function is the critical piece. By hashing the flag name and user ID together with SHA-256 and taking the result modulo 100, we get a stable integer in [0, 99] for each (flag, user) pair. Set rollout_pct to 10, and the 10% of users whose bucket falls in [0, 9] always see the flag enabled — consistently, across requests, across sessions, across restarts.
Go Implementation
|
|
TypeScript/Node Implementation
|
|
When Simple Is Enough
A toggle table works well when:
- You have fewer than ~50 active flags
- You don’t need per-flag audit logs or a management UI
- Your team is small enough that everyone knows what flags exist
- You have privacy constraints that prevent sending evaluation data to third-party SaaS
When you outgrow it: no audit log, no activation strategies, no SDK ecosystem, no metrics on how often each flag is evaluated. That is when you reach for Unleash or LaunchDarkly.
Unleash: The Open-Source Feature Flag Platform
Unleash is a mature, self-hostable open-source feature flag service. It has been in production since 2014, has SDKs for every major language and framework, and provides everything a toggle table lacks: a management UI, audit logging, activation strategies, and impression metrics.
Self-Hosting with Docker Compose
|
|
Run docker compose up -d and the Unleash UI is available at http://localhost:4242. Default credentials: admin / unleash4all.
Unleash uses a polling architecture: SDK clients periodically fetch the current flag configuration (every 15 seconds by default) and cache it locally. Flag evaluation happens entirely in process — no network request per evaluation. This means sub-millisecond flag checks and resilience to Unleash server outages.
Activation Strategies
Activation strategies are Unleash’s killer feature. Each flag can have multiple strategies, any of which can enable the flag for a given request context.
Default (on/off) — Simple boolean. Enable for everyone or nobody.
UserIDs — Provide a comma-separated list of user IDs. The flag is enabled only for those exact users. Perfect for internal dogfooding.
Gradual Rollout — Percentage-based with consistent hashing. Set to 10% and exactly 10% of your users see the feature. Increment it over time as confidence grows. Unleash hashes by userId, sessionId, or random — choose userId for persistent per-user assignment.
IPs — Enable for requests from specific IP addresses or CIDR ranges. Useful for office-only features or staging environments.
Custom Strategies — Define your own context fields and evaluation logic. Examples: “enable for accounts on the Enterprise plan”, “enable in the EU region”, “enable for accounts created before 2025-01-01”.
The Unleash Context
Every flag evaluation receives a context object. The context is what activation strategies use to make decisions:
|
|
Populate the context with as much information as you can — you will want to target flags by plan or region later, and if that data is not in the context, you cannot use it in activation strategies without a redeploy.
SDK Examples
Python:
|
|
Node.js:
|
|
Go:
|
|
Feature Flag Variants
Unleash supports variants, which extend flags from boolean to multi-value. Instead of just on/off, a flag can return different payloads per variant — the foundation of proper A/B testing.
|
|
Variants have configurable weights (e.g., 33%/33%/34% for three variants) and support payload types: strings, JSON, CSV, or numbers. Unleash tracks impression data per variant — how many times each was served — giving you the raw numbers for your experiment analysis.
LaunchDarkly: The Enterprise Standard
LaunchDarkly is the hosted feature flag platform and market leader. Where Unleash is self-hosted open source, LaunchDarkly is managed SaaS with a commercial model.
Pricing starts at approximately $10/seat/month, with a free tier supporting up to 1,000 Monthly Active Users. For teams that do not want to operate their own flag infrastructure — and can accept sending flag evaluation context to a third party — LaunchDarkly removes all operational overhead.
What Makes LaunchDarkly Different
Targeting rules — LaunchDarkly’s rule builder is the most powerful in the market. You can write rules like: “enable for users where plan = 'enterprise' AND country IN ['US', 'CA', 'GB']” with AND/OR logic, string contains/starts with, numeric comparisons, and semantic versioning comparisons. No custom strategy code required.
Experiments — Built-in A/B testing with statistical significance tracking. LaunchDarkly calculates p-values and confidence intervals from impression and conversion data, flagging when results reach significance.
Data Export — Stream all flag evaluation events to your data warehouse (Snowflake, BigQuery, Redshift) for custom analysis.
Integrations — Native integrations with Jira (link flags to tickets), Slack (flag change notifications), Datadog (flag change events overlaid on metrics), PagerDuty, and dozens more.
SDK streaming — LaunchDarkly SDKs maintain a persistent SSE connection to the LaunchDarkly streaming service. Flag changes propagate in milliseconds rather than polling intervals.
Python SDK Setup
|
|
The second argument to variation is the default value — returned if the SDK has not initialized, the flag does not exist, or there is any evaluation error. This default is your safety net: always choose the safe, proven behavior.
Targeting Rules in Practice
LaunchDarkly evaluates rules top-down, returning the first matching rule’s variation. A typical production flag might look like:
- If
emailends with@yourcompany.com→ Servetrue(internal dogfood) - If
planis one of["enterprise", "business"]→ Servetrue(paying customers first) - If
userIdis in rollout group (15% gradual rollout) → Servetrue - Default rule → Serve
false
This maps naturally to a real progressive rollout: internal team first, then paying customers, then gradual expansion to free users.
LaunchDarkly vs Unleash
| Concern | Unleash (self-hosted) | LaunchDarkly (SaaS) |
|---|---|---|
| Cost | Hosting + ops time | ~$10/seat/month |
| Data privacy | All data stays in your infra | Evaluation context sent to LD |
| Ops overhead | You maintain the service | Zero |
| Targeting power | Good (custom strategies) | Excellent (visual rule builder) |
| Experiment analysis | Basic impression counts | Full statistical significance |
| SDK quality | Good, community-maintained | Excellent, first-party |
| Streaming updates | Polling (15s default) | SSE (milliseconds) |
Use Unleash when: you have ops capacity, privacy requirements prevent third-party data sharing, or budget is tight. Use LaunchDarkly when: you want a fully managed service, need rich targeting rules without custom code, or are running rigorous product experiments.
Implementation Patterns in Real Applications
Where to Check Flags
Check flags at the boundary of your application — in the HTTP handler, the queue consumer, the scheduled job entry point — not deep in business logic or utility functions.
The closer a flag check is to user-facing code, the clearer it is what behavior it controls. A flag buried five layers deep in a utility function is a flag nobody understands.
|
|
The Flag Wrapper Pattern
Do not scatter is_enabled("flag-name-as-string") calls throughout your codebase. String literals are invisible to your IDE, refactoring tools, and static analysis. Instead, wrap flag checks behind named functions:
|
|
The wrapper is searchable, documents its own removal date, and can be updated in one place if the flag name changes.
Backend vs Frontend Flags
Server-side evaluation is more secure — users cannot inspect flag rules or force themselves into a variant by manipulating client-side state. Server-rendered content reflects flag state before the HTML leaves the server. Use server-side evaluation for security-sensitive flags, pricing changes, and access control.
Client-side evaluation moves flag logic to the browser or mobile app. LaunchDarkly and Unleash both have frontend SDKs that bootstrap flag state on page load. This enables faster rendering and client-side A/B tests, but flag rules must be safe to expose publicly.
For SPAs, the classic problem is a flash of wrong content: the page renders with the old experience for a moment before the flag SDK initializes and updates the UI. The fix is to bootstrap initial flag state into the server-rendered HTML:
|
|
The frontend SDK reads window.__FEATURE_FLAGS__ immediately, then subscribes to streaming updates for changes.
Testing with Flags
Never use a real flag service in unit tests. Inject a mock evaluator:
|
|
Test both states of every flag. It is a common failure mode to test only the happy path (flag on) and discover the flag-off path is broken only when you try to roll back.
Avoiding Flag Spaghetti
Never check a flag inside another flag’s code branch:
|
|
Nested flags create exponential test matrices and make behavior nearly impossible to reason about. If two features always go together, they should share one flag. If they are independent, they belong at the same level of the code.
Operational Patterns: The Lifecycle of a Flag
Naming Conventions
Consistent naming makes flags searchable and self-documenting. Two common conventions:
feature-<team>-<description>:feature-checkout-new-payment-flowrelease-<ticket-id>-<description>:release-eng4421-new-checkout-redesign
Prefix ops flags distinctly: ops-rate-limit-upstream-api, ops-circuit-breaker-payments.
Progressive Rollout Procedure
The standard progression for a release flag:
- Deploy with flag off — Code ships to production. Zero users affected. Verify the deploy is healthy.
- Enable for internal users — Flag on for employees. 10-50 users. Find obvious bugs.
- 1% rollout — Watch error rates, latency p99, and business metrics for 30-60 minutes.
- 5% rollout — Statistically meaningful signal. Watch for another hour.
- 25% rollout — At this point you are fairly confident. Verify business metrics are not degrading.
- 50% rollout — Roughly even split. Compare old vs new directly.
- 100% rollout — Everyone on the new path.
- Delete old code path — Remove the conditional and the legacy branch.
- Delete the flag — Remove from the database/platform and any SDKs.
Each step should have an explicit green light from your metrics. If anything looks wrong at any stage, drop back to the previous percentage immediately — no deploy required.
The Kill Switch Procedure
A kill switch flag is a permanent ops flag used as a circuit breaker. When you see error rates spiking:
- Open your feature flag management interface (bookmarked, accessible without VPN if possible)
- Toggle the flag off — behavior reverts to safe baseline in seconds
- Write a brief incident note: when you toggled, what changed, what you observed
- Investigate root cause with the system stable
The kill switch is only useful if you have rehearsed using it. Make sure every on-call engineer knows where flags live and how to toggle them.
Flag Cleanup
Schedule monthly flag reviews. Any release or experiment flag older than 90 days with rollout_pct < 100 is a problem: either complete the rollout or revert and delete the flag. A flag that exists at 50% for three months means half your users have a different experience indefinitely — that is flag debt turned into technical debt.
Automate this with a cron job that queries your flags table for old release flags and posts a Slack message to the owning team.
Monitoring Flag Changes
Every flag change in production should generate a notification. Unleash and LaunchDarkly both have audit logs. Wire them to your monitoring stack:
- Datadog: LaunchDarkly’s integration adds flag changes as events overlaid on your dashboards — instantly correlate a metric spike with a flag toggle.
- Slack: Webhook on flag changes →
#ops-flagschannel. The on-call engineer sees it in real time. - PagerDuty: For kill switch toggles, alert the team immediately.
Advanced Patterns
Dark Launching
Dark launching runs new code in production alongside old code, but only returns the old code’s results to users. The new code executes silently, its output compared for correctness and performance in the background.
|
|
Dark launching is how you validate a new algorithm or service against production load before any user experiences it. It is particularly valuable for search ranking changes, recommendation engines, and pricing logic.
Database Migrations with Flags
Backwards-incompatible database migrations combined with feature flags enable zero-downtime schema changes:
- Deploy migration — Add the new column (nullable, no default required). Old code ignores it.
- Backfill — Populate the new column for existing rows asynchronously.
- Deploy code with flag — New code path reads from the new column; old path reads from old column. Flag off.
- Ramp up flag — Gradually shift traffic to the new schema path, verifying data integrity.
- 100% flag — All traffic uses the new schema.
- Remove old column and code — Clean up once confident.
This decomposition means a schema change that would previously require a maintenance window can be done with zero downtime and full rollback capability at each step.
Feature Flags in Microservices
When a flag gates behavior that spans multiple services, you need to propagate flag evaluation results across service boundaries. The two approaches:
Evaluate once at the edge — The API gateway or BFF evaluates all flags for a request and injects results into downstream HTTP headers or gRPC metadata: X-Feature-New-Checkout: true. Downstream services read the header rather than evaluating the flag themselves. This is simpler and ensures consistent behavior within a single request.
Each service evaluates independently — Every service has the flag SDK and evaluates flags against the same user context. This works if your flag platform has low enough latency (in-process evaluation like LaunchDarkly SDKs), but requires that each service has access to user context.
OpenFeature: The CNCF Standard
OpenFeature is a CNCF project defining a vendor-neutral interface for feature flag SDKs. Instead of writing code against LaunchDarkly’s SDK directly, you write against the OpenFeature interface — then swap the provider (LaunchDarkly, Unleash, your toggle table) without changing application code.
|
|
OpenFeature is worth considering for any new greenfield project. Vendor lock-in on feature flags is real — switching from LaunchDarkly to Unleash after three years of ldclient.variation() calls is painful. OpenFeature eliminates that migration cost.
Pitfalls and Anti-Patterns
Flag explosion — Teams create flags for every change, and within a year the system has 500 active flags that nobody fully understands. Establish a flag creation process: document owner, type, and removal date. Run quarterly audits.
Testing only the happy path — You test the flag-on state thoroughly and forget to test flag-off. A production incident reveals that the legacy code path has been silently broken for weeks. Always run both paths through CI.
Long-lived release flags — A release flag that has been at 50% for four months is a liability. It means your codebase has two active production code paths with diverging test coverage. Set a calendar reminder at flag creation for 90 days: “complete rollout or revert.”
Flags in the wrong layer — Feature flags do not belong in database repositories, utility libraries, or deep domain services. They belong at the edges of your system. Putting them deep in shared code means you cannot test or reason about the behavior from the outside.
The stale flag problem — An SDK checks a flag that no longer exists in the configuration store. Always return a safe default. Never throw an error or return an ambiguous null — make the default the safe, backwards-compatible behavior:
|
|
Skipping caching — Checking a database on every HTTP request for flag state will crater performance. Caching is not optional — it is the mechanism that makes the whole system fast. Use a 30-60 second TTL in Redis or an in-memory LRU cache, and invalidate on flag changes via webhooks.
No polling or streaming — If your flag service is unreachable and you have no local cache of flag state, every evaluation fails. Ensure your SDK has an in-memory snapshot of the last known state, so a brief flag service outage does not take down your application.
Summary
Feature flags are one of the highest-leverage operational tools available to a software team. The pattern is simple — a conditional backed by runtime configuration — but applied consistently it transforms how you deploy and release software.
Start with a toggle table if your needs are modest. Self-host Unleash when you need a management UI, audit log, and activation strategies without a SaaS bill. Adopt LaunchDarkly when you need enterprise targeting, statistical experiments, and fully managed infrastructure.
Regardless of which platform you choose, the discipline matters more than the tool:
- Set removal dates when creating release and experiment flags
- Check flags at the boundaries of your system, not deep in shared code
- Test both states of every flag in CI
- Monitor flag changes as operational events
- Run progressive rollouts — 1%, 5%, 25%, 50%, 100% — with observability at each step
Deploy constantly. Release carefully. Flags give you the ability to do both.
Comments