Testing in production is safest when it is a controlled, time-limited canary—not a replacement for automated testing. Deploy a small change to a representative slice of real traffic, compare user-facing health signals with a control, and expand exposure only when the evidence supports it. Feature flags can keep feature exposure separate from code deployment, but they do not replace monitoring or a tested rollback path.
What production testing, canaries, and feature flags mean
Continuous delivery is not the same as continuous deployment
DORA defines continuous delivery as keeping software deployable so teams can release changes quickly, safely, and sustainably when they choose. It does not mean every change must automatically reach every user. Continuous deployment goes further: the practice is to try to deploy each code change to production as soon as possible. A team can adopt continuous delivery without choosing continuous deployment.
As an Amazon Associate I earn from qualifying purchases.
A canary exposes a change to a limited share of production
A canary is a partial, time-limited deployment of a change followed by evaluation. The canary receives the change while the rest of the service provides a control for comparison. Real traffic can reveal problems missed by unit or load tests, but people in the canary group are real users and may be affected. Google SRE therefore treats canarying as a controlled production test, not proof that a release is safe.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →A feature flag controls exposure, not whether code was deployed
A feature flag separates a feature’s launch or configuration from the release of the binary that contains it. A team can deploy code with a feature disabled, then enable it for a selected cohort. If the feature behaves unexpectedly, the team may be able to disable it without waiting for another binary deployment. That depends on the system’s flag architecture and operational controls; a flag does not undo every code change or eliminate the need for rollback.
#1 Best Overall
How to run a production canary
- Keep preproduction safeguards in the pipeline. Continue automated tests, security checks, and build and deployment checks. DORA’s continuous-delivery practices include test and deployment automation, as well as comprehensive testing and observability.
- Choose a small, self-contained change. Deploy it behind a flag or route a limited traffic cohort to the changed version. Use a flag for independent feature exposure where the system supports it; use deployment controls to determine which version receives traffic.
- Set the control and decision criteria before exposure. Select a small number of signals tied to user-visible service health—such as errors and latency—and decide what result would pause or stop the rollout. Ensure the canary and control can be compared meaningfully, rather than attributing unrelated differences to the change.
- Choose a representative sample and sufficient observation window. The canary needs enough traffic and time to encounter relevant behavior. There is no universal safe percentage or duration: traffic volume, workload diversity, metric behavior, and release cadence all affect what is informative.
- Pause, investigate, or reverse when signals worsen. Depending on the cause and available controls, disable the feature, roll back the release, or cancel the rollout. Make the rollback path operationally safe before relying on it. Promote exposure in stages only when the evidence supports doing so.
- Keep monitoring after promotion. Some defects are delayed or depend on specific circumstances. Google Cloud’s change guidance emphasizes continued monitoring because not every issue appears during the initial rollout.
What makes a canary result trustworthy
- Relevant signals: Track measures connected to service health for users, not just whether the deployment completed. Google SRE discusses errors and latency as useful examples.
- Fair comparison: Compare like with like. If the canary handles a different workload or user mix from the control, a metric difference may not be caused by the release.
- Sensible thresholds: Thresholds that are too strict can trigger misleading stops; thresholds that are too loose can miss harm. Choose criteria appropriate to the service and its normal variation.
- Enough exposure: A tiny or brief sample may fail to encounter representative inputs. Conversely, a canary is not harmless merely because it is limited: affected users can still experience a defect.
- Actionable gates: Staged traffic without predefined evaluation criteria is not, by itself, a meaningful safety check. Decide who or what evaluates the signals and what happens when they cross a limit.
As Google SRE explains, the defining value of a canary is that it handles real user traffic; that is also why it should be treated as an early, controlled deployment step rather than the final stage of testing. Adrian Hilton’s Google Cloud article on release canaries makes the same point while noting that real users can be affected.
Keep feature exposure control separate from traffic control
Flags and canaries solve related but different problems. A flag determines whether a feature is active for a user or cohort; a deployment strategy determines which software version handles traffic. Evaluate both layers instead of assuming one replaces the other.
| Decision | Feature flag | Canary or deployment strategy |
|---|---|---|
| What it controls | Whether a feature is enabled, potentially by cohort or rule | Which version receives some or all production traffic |
| Typical use | Enable a deployed feature selectively, or disable it without waiting for another binary where supported | Expose a new version to a portion of traffic, evaluate it, then promote or reverse the rollout |
| Key safety question | Can the feature be targeted and switched off reliably? | Can the canary be compared fairly with control and stopped or rolled back safely? |
When choosing an implementation, consider cohort representativeness, whether health signals can be attributed to the change, rollout duration and promotion gates, the speed and safety of disabling or rolling back, operational complexity, and fit with the platform and runtime. Google Cloud Deploy documents staged deployment strategies; it is one platform-specific example, not a universal recommendation.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsPlatform-specific note: Cloud Run canary behavior
Google Cloud Deploy’s Cloud Run canary guidance describes progressive traffic splitting. It also specifies an important first-deployment exception: when deploying to a target for the first time, the process skips canary phases because there is no existing version with which to split traffic. That behavior is specific to this documented implementation; check the controls and first-deployment behavior of the platform you use.
Quick Recap
Common failure modes to avoid
- Treating a flag as a safety net for everything: It may disable a feature, but it does not replace automated tests, service monitoring, or a way to reverse a problematic version.
- Promoting by schedule rather than evidence: A rollout should advance because its agreed health criteria are met, not simply because a fixed interval has passed.
- Using metrics that cannot distinguish the release’s effect: If canary and control populations differ materially, investigate that difference before drawing a conclusion.
- Stopping observation immediately after rollout: A clean early canary cannot rule out delayed or circumstance-specific failures.
- Assuming one canary size works for every service: The cited guidance provides no universal traffic percentage or duration; choose based on the workload and the quality of evidence needed.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
