The team merges a refactor of the payment flow. CI passes. The deploy pipeline runs. Five minutes later, every user trying to check out hits a 500 error. Ten minutes after that, the team has rolled back, but $20k of orders are lost and customer support is dealing with refund requests. The bug only manifests under real production traffic; load testing didn’t catch it because the failure is on a specific edge case.
The fix is not “test more carefully.” Some bugs only surface in production, with real users. The fix is to limit the blast radius of the inevitable bad deploy by rolling out gradually: 1% of users first, then 10%, then 50%, then 100%. A bug at 1% is a small fire. A bug at 100% is a postmortem.
This post is the five-stage rollout pattern, the metrics that gate each stage, the tooling, and the rollback procedure that has to actually work.
Stage 0: Internal users only (5-10 employees, typically 0-1 days)
Stage 1: 1% of production traffic (1-2 days)
Stage 2: 10% of production traffic (1-2 days)
Stage 3: 50% of production traffic (1-2 days)
Stage 4: 100% of production traffic (the feature is launched)
Cleanup: Remove the flag and the old code pathThe total time from first deploy to 100% is typically 4-7 days for a non-critical feature, longer for high-risk changes (payments, auth).
The mechanism is a feature flag. Code path A is the old behavior, code path B is the new. The flag percentage controls how many users get B.
if (await flags.isEnabled('new-checkout-flow', { user })) {
return renderNewCheckout();
}
return renderOldCheckout();Both code paths are in production simultaneously. The flag is what selects which one runs.
Don’t promote stages on a timer alone. Gate them on observed metrics:
Stage 0 → 1: Internal usage shows no errors for 24 hours. The team has manually tested the happy path and a few edge cases.
Stage 1 → 2: At 1% for 24-48 hours:
Stage 2 → 3: Same checks at 10%. Plus: any rare failure modes that need volume to surface have had time to appear.
Stage 3 → 4: Same at 50%. By this point, you have high confidence the change is good.
Cleanup: After 100% holds for 1 week, remove the old code and the flag (see the feature-flags post).
A small set covers most use cases:
Compare each metric for users in the new variant vs users in the old variant, not absolute. A 10% error rate is bad universally; the question is “did this change make it worse?”
Three options:
1. Random by user ID. Stable bucketing: a user who got the new flow on Monday gets it on Tuesday too. The right default. Use a hash of (user_id, flag_name).
2. By cohort. Internal employees first, then beta opt-ins, then specific customer segments, then everyone.
3. By geographic region. Deploy to one region first to limit blast radius further.
For most rollouts, (1) is sufficient. (2) and (3) layer on top for extra-risky changes.
A rollout is only as good as its rollback. The procedure must be:
The single command:
flagsctl set new-checkout-flow --percentage 0Or click a button in LaunchDarkly. Either way, sub-minute. Document the runbook, including who has access.
For most teams, buying a feature-flag service is the right call:
For very small teams, a Postgres-backed flag table works:
CREATE TABLE feature_flags (
name text PRIMARY KEY,
percentage int NOT NULL DEFAULT 0,
updated_at timestamptz NOT NULL DEFAULT now()
);async function isEnabled(name: string, user: { id: string }): Promise<boolean> {
const { rows } = await pool.query(
'SELECT percentage FROM feature_flags WHERE name = $1', [name]);
if (!rows[0]) return false;
const hash = murmurHash(user.id + name) % 100;
return hash < rows[0].percentage;
}Add caching (30s TTL) and you have a working flag system in 30 lines. For under ~20 flags, this is fine. Past that, buy.
1. Sticky bucketing not implemented. A user sees the old flow on one request and the new flow on the next. Confusing UX, broken UX in some cases. Always bucket by (user_id, flag_name) hash, deterministic.
2. The new and old paths share state in incompatible ways. A new flow writes data the old flow can’t read. When you roll back, users on the new flow are stranded. Design changes so old and new are mutually compatible.
3. Comparing metrics globally instead of per-variant. Total error rate is at 0.5%; you don’t notice that error rate for new-flow users is 5%. Always slice metrics by variant.
4. Skipping stages under deadline pressure. “We have to ship by Friday, let’s go straight to 50%.” That is exactly the situation that produces the postmortem. Stages exist to prevent the rare bad outcome; the rare bad outcome is exactly when you’d be tempted to skip them.
5. Forgetting to clean up. The flag is at 100% for 6 months and the old code is still in the codebase. Set a calendar reminder to delete the flag two weeks after 100%.
For very risky changes (database migrations, major refactors), “dark launch” first:
For example:
async function chargeCustomer(orderId: string) {
const oldResult = await chargeCustomerOldFlow(orderId);
if (await flags.isEnabled('dark-launch-new-charge')) {
try {
const newResult = await chargeCustomerNewFlowDryRun(orderId);
logComparison({ orderId, old: oldResult, new: newResult });
} catch (err) {
logDarkLaunchError({ orderId, err });
}
}
return oldResult;
}Production behavior is unchanged. New code is exercising real data. Discrepancies surface without affecting users.
For the most critical paths (payment, auth), more granular stages help:
Internal → 0.1% → 1% → 5% → 10% → 25% → 50% → 100%Eight stages, each at the prior level for 24-48 hours. A change that passes all of these is genuinely battle-tested.
Conversely, for trivial changes (a copy update, a bug fix in non-critical code), three stages may be enough:
Internal → 50% → 100%Match the rollout granularity to the risk.
A staged rollout converts inevitable bugs from disasters into observations. 1%, 10%, 50%, 100%, each gated on metrics, with a fast rollback. Bucket users stably. Compare per-variant metrics, not totals. Clean up flags after they’ve been at 100%.
The team that adopts this finds that “we shipped a bad deploy” stops meaning “all customers were affected” and starts meaning “1% of customers saw a transient bug for two hours.” That is the difference between a good engineering culture and a fire-fighting one.
The kind of release-engineering discipline that turns “we hope this works” into “we measured at each stage and it worked” (staged rollouts, rollback procedures, per-variant metrics) is the kind of long-haul engineering practice Yojji’s teams build into the products they ship for clients.
Yojji is an international custom software development company founded in 2016, with teams across Europe, the US, and the UK. They specialize in the JavaScript ecosystem (React, Node.js, TypeScript), cloud platforms (AWS, Azure, GCP), and full-cycle product engineering, including the rollout and deployment practices that decide whether shipping is risky or routine.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。