Hardware FixRecommendedDevice not working? Your driver may be the problemCheck updates for common hardware issues.Fix DriversOctober DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsWindows FixRecommendedWindows errors stealing your time? Find the fix fastScan stability, cleanup and performance issues.Fix Now×
Skip to content
World desk6 min

How to Roll Back a Bad Software Deployment: Take a Bad Change Back Before Anyone Notices

Learn how to take a bad software change back: git revert for committed code, platform rollback for deployments, feature flags for shipped behavior, and canary rollouts that limit exposure.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

To take a bad change back, you first stop it from spreading, then pick the right recovery tool for where the change lives: committed source, a running deployment, or a feature flag. Good source history, staged exposure, monitoring, and a rehearsed recovery path cannot guarantee that nobody notices. What they do is limit how many users are affected and shorten the time to recovery.

Three recovery tools that solve different problems

People often use “rollback” for three separate actions. They overlap, but each one repairs a different layer of the system.

As an Amazon Associate I earn from qualifying purchases.

Recovery tool What it reverses What it does not do
Source revert (git revert) A committed change, by recording a new commit that undoes it Does not redeploy anything or undo data, infrastructure, or artifacts produced by the bad build
Deployment rollback The version that is running in an environment, by redeploying an earlier known-good version through the platform’s procedure Does not rewrite history, and does not automatically return database state or generated artifacts to a safe state
Feature disablement The behavior of new code that already shipped, by turning a flag off Does not remove the code, and does not help if the fault lives in a change that is not behind a flag

In practice, containment usually comes first (stop exposure or disable the feature), then the team chooses between reverting the source and rolling back the deployment, or fixing forward.

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Prepare before the change ships

Most of the work that determines how fast you recover happens before the bad change reaches production. Three preparations matter most.

  • Keep changes small enough to reason about. A small commit is easier to identify, revert, and explain than a bundled release.
  • Deploy through an approved, repeatable workflow. If the only way to deploy is a manual sequence that exists in one engineer’s memory, rollback will be a manual sequence too.
  • Define the stop signal in advance. Decide which health indicators (error rate, latency, a key business transaction) will halt the rollout, and who is allowed to halt it.

The AWS Well-Architected Framework, in its operational practice OPS06-BP01, states: “In either situation, a fix forward or rollback plan should be well documented and tested before deployment to live production so that the time it takes to revert a change is minimized.” The guidance also recommends measuring outage duration and change data so that the next recovery is faster than the last.

Microsoft’s Azure guidance on safe deployments takes the same line on stopping: when a rollout group reports an issue, the rollout should stop immediately, and the team then investigates cause and severity. The point is to stop first and diagnose second.

Revert a bad commit in Git

Use git revert when the intent is to record a compensating change in the history rather than erase the bad one. Git’s documentation describes it as creating a new commit that reverses an earlier commit, which keeps the record of both the change and the correction. Reset and restore are different: Git’s manual warns that those commands can discard uncommitted changes, so they are not a safe substitute for a revert on shared history.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  1. Identify the faulty commit with git log and confirm its scope with git show <commit>.
  2. Confirm the working tree is clean with git status. Git requires a clean working tree for revert, so stash or commit anything in progress first.
  3. Run git revert <commit>. Git opens the commit message editor; write a message that states the bad change, the symptom, and why the revert is the chosen fix.
  4. If the commit is a merge, add -m 1 (or the parent number of the mainline) so Git knows which side to keep. Understand that reverting a merge changes what later merges bring in. Reverting a merge does not remove the changes from the branch that was merged, so a later merge of that branch may not reapply them until the situation is handled deliberately.
  5. If conflicts occur, resolve them, then run git revert --continue. Use git revert --skip to drop the current commit from the sequence, or git revert --abort to return the branch to its state before the revert started.
  6. Push the revert through the normal review and pipeline path. Reverting in Git changes source only; it does not change what is running in production until a deployment carries it out.

Roll back a deployed version

When the bad version is already in an environment, first identify the version that is actually running. The commit you think is live and the artifact that is live are not always the same. Then run the rollback procedure your deployment platform defines. Do not assume that returning the application code also returns data, infrastructure, configuration, or generated artifacts to a safe state.

Job-based rollbacks in GitLab

GitLab’s deployment documentation describes a rollback as creating a new deployment that points to an earlier commit. The deployment script must define how that deployment happens, so a rollback only works if the rollback logic is present in the job script. GitLab also notes that some artifact-producing jobs from earlier pipelines may need to be run again separately, because the earlier build outputs are not necessarily what the rollback needs. Check this during rehearsal, not during the incident.

Rolling back with stateful changes

Rollback is hardest when the bad change touched a database, a schema, a queue, or another stateful system. If the new version wrote data in a format the old version cannot read, returning the code can make the failure worse. In that situation a safe fix-forward may be the better path, or the rollback plan must include the data steps explicitly. Azure’s guidance on safe deployments makes the same point: the recovery procedure has to account for data changes and workload design.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Use staged exposure to limit the damage

Staged rollouts reduce how many users see a bad change before anyone can act. Canary, rolling, blue/green, traffic splitting, and feature flags all provide a way to limit exposure or to halt or reverse a rollout. The canary is the clearest example: the new version runs beside the stable deployment and receives a small share of traffic, and the team watches errors, performance, and unexpected behavior before increasing that share.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Canary in Kubernetes

Kubernetes’ canary tutorial shows the basic pattern. The canary Deployment runs alongside the stable Deployment, and the team can stop the canary by scaling it to zero with a command such as kubectl scale deployment <canary-name> --replicas=0. Scaling to zero keeps the Deployment object in place, so it can still be inspected after the fact, and stable capacity can be restored as needed. Production systems usually automate the sequence: shift traffic, monitor, then promote or roll back based on the health signals defined in advance.

Choose between rollback and fix-forward

Rollback returns the system to a known working configuration. Fix-forward applies a correction during the rollout. Neither is universally better, and the choice depends on three things: how severe the impact is, what state the system is in, and whether any stateful change can be reversed safely.

  • Roll back when the previous version is known to work, the change is recent and self-contained, and reversing it does not require undoing data writes.
  • Fix forward when the previous version cannot safely read the new data, when the fix is small and well understood, or when rolling back would itself cause an outage.
  • Disable the feature first, when the new behavior sits behind a flag. This contains the impact quickly and leaves the decision about code for later.

Recovery checklist

  • Identify the faulty change and its scope: which commit, which artifact, which environments, and which users.
  • Stop or limit further rollout, and disable the feature if a flag controls it.
  • Choose rollback or fix-forward based on the system and data state.
  • Verify the restored behavior with monitoring, not only with a successful deployment status.
  • Communicate the incident and record the reason for the chosen recovery path.
  • Review the recovery time and update the procedure so the next recovery is faster.

No single statistic on rollback time or deployment failure rates is established across the official guidance cited here, so treat the recovery time you measure in your own system as the baseline that matters.

“

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
PC Slower Than It Used to Be?Free scan - under a minute
Outdated Drivers Are Slowing You DownFree scan - exact matches

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.