Hardware FixRecommendedDevice not working? Your driver may be the problemCheck updates for common hardware issues.Fix DriversHalloween DealsAmazon USHalloween week deals: useful picks before month-endAmazon US: current deals, useful picks and tech finds.Check DealsClean PCRecommendedOne scan can reveal what keeps slowing WindowsLook for cleanup and repair opportunities.Run Scan×
FREEDOM251

Freedom251 › Blog › Argo CD

How to troubleshoot drift and failed promotions across Kubernetes clusters

By Freedom251 Team · · 11 min read

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Start by identifying which layer is failing: declared Git state, GitOps reconciliation, or workload health. An Argo CD OutOfSync status means the live resources differ from the rendered target; it does not prove that a Deployment rollout failed. A stalled Deployment is a Kubernetes workload-controller problem, even when the promotion controller reports a successful sync.

Record the cluster, namespace, application, resource name, Git revision or image, promotion stage, and first-failure time. Then compare the failing cluster with a known-good cluster using the same fields. This prevents repeated sync attempts from masking a quota, permission, image, admission, configuration, or probe difference.

As an Amazon Associate I earn from qualifying purchases.

Separate drift, sync failure, and rollout failure

Use the component reporting the symptom as your starting point. Argo CD defines an application whose live state differs from its target state as OutOfSync. Kubernetes reports Deployment progress through conditions, ReplicaSet availability, events, and rollout status. These signals can occur together, but they describe different failures.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Observed symptom What it establishes Inspect next
Argo CD shows OutOfSync At least one live resource differs from the rendered desired state. Rendered target, live diff, sync operation, and resource ownership.
Argo CD sync failed The controller did not complete the requested apply operation. Sync error, revision, parameters, retry policy, and admission or permission errors.
Application is synced but unhealthy The declared resources may exist, but one or more resources are not healthy. Deployment conditions, Pods, events, probes, and dependent services.
kubectl rollout status does not complete The Deployment has not met its progress criteria for the observed revision. ReplicaSet, Pod scheduling, image pulls, readiness, quota, and ProgressDeadlineExceeded.
Promotion is green in one cluster and fails in another The clusters differ in inputs, controller state, or runtime conditions. Same revision and rendered manifests first, then environment-specific constraints.

Do not treat a green promotion status as proof that users can reach healthy Pods. A promotion controller can finish its own workflow while a Deployment remains unavailable or while readiness probes keep every new Pod out of Service endpoints.

#1 Best Overall
Sale
TOZO A1 Wireless Earbuds Bluetooth 5.3 Light Weight in Ear Headphones | IPX5 Waterproof,2 Mics for AI Calls,Immersive Premium Sound Bass Headset with Charging Case,32 EQ via App Black
  • [Ultra-Lightweight Ear Buds Designed for Small Ears] Each earbud weighs only 3.7g and features a compact, ergonomic in-ear design made especially for small ears. Secure, low-profile, and comfortable for workouts, all-day wear or overnight sleeping.The ultra-slim profile sits flush in your ear with no hard edges poking or rubbing, so you can even sleep on your side without discomfort.
  • [Immersive Stereo Sound with TOZO OrigX Technology] TOZO OrigX tuning delivers clear vocals, balanced mids, and natural stereo sound for music, podcasts, and videos.
  • [Long Battery Life for Daily Use] Get up to 7 hours of playtime on a single charge, with up to 32 hours total using the charging case—ideal for workdays, commuting, and extended listening sessions.
  • [Bluetooth 5.3 & Stable Connection] Bluetooth 5.3 provides fast pairing, stable wireless performance, and reduced dropouts as you move around home or office.
  • [Deep Bass with Clear Vocals] High-performance drivers produce punchy bass while keeping vocals clean and detailed for everyday listening.

Build an incident baseline before changing anything

Capture evidence from both the failing and known-good clusters before retrying, scaling, or rolling back. Include the exact time zone and timestamps so events can be correlated with controller logs and monitoring.

  1. Identify the scope. Write down cluster context, namespace, Argo CD application, Deployment and ReplicaSet names, promotion stage, Git commit, image digest or tag, and the first observed failure.
  2. Save controller state. Record Argo CD application sync status, health status, operation result, rendered revision, parameters, retry settings, and whether automated sync or self-heal is enabled.
  3. Save workload state. Capture Deployment conditions, desired/current/ready replica counts, the active ReplicaSet revision, Pod phases, container states, and recent events.
  4. Compare inputs. Diff the rendered manifests and Helm or Kustomize inputs used for each cluster. A different values file, overlay, image digest, namespace, or admission mutation can explain a cluster-specific result even when the Git commit is identical.
  5. Preserve failed resources. Keep the failing ReplicaSet and Pods long enough to inspect logs, events, probes, and environment variables. Deleting them too early removes the most useful evidence.

Why an app can remain OutOfSync after syncing

Inspect the rendered target and live diff

First confirm what Argo CD rendered, not merely what is present in Git. In the application view, open the resource tree and diff for the affected object. Check fields that are commonly changed after apply: labels, annotations, replica counts, generated names, admission-injected fields, and controller-managed status. A persistent difference can mean the desired manifest is wrong, another controller is mutating the object, or Argo CD is not managing the object you think it is.

On the cluster, use read-only commands to confirm the live object and its owner:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
kubectl get deployment <name> -n <namespace> -o yaml
kubectl describe deployment <name> -n <namespace>
kubectl get rs,pods -n <namespace> --show-labels

Compare the output with the rendered target, while excluding fields that are intentionally generated or managed by Kubernetes. If a field changes immediately after every sync, identify the mutating controller or admission policy before changing the manifest.

Check the sync operation separately from application status

An application may show a completed sync operation while one resource is still unhealthy, or show OutOfSync because a later mutation changed live state. Review the operation result, resource-level messages, revision, and parameters. A failed apply should lead to the exact API error; a successful apply followed by drift calls for an ownership or mutation investigation rather than another blind sync.

Rank #2
Sale
Smart Watch for Men Women Compatible with iPhone Samsung Android Phone 1.69 inch Full Touch Screen IP68 Waterproof Bluetooh Fitness Tracker Heart Rate/Sleep Monitor (Black)
  • 【IP68 Waterproof Multifunctinal Smart Watch】Torjalph fitness activity tracker smart watch is IP68 waterproof technology ,you can wear it when swimming,handwashing,it's rain-proof which means that you perform multiple exercise modes outside without worrying about sweat or rain corroding your fitness smart watch,and this watch equipped with a 230 mAh large capacity battery magnetic charging ,and this sports activity tracker watch can be used 5-7 days If fully charged.
  • 【Larger 1.83'' Full Screen Watch & Multi-dial Selection】1.83 inch full HD color touch screen smart watch for men / women provides excellent 240*284 resolution image quality ,This Bluetooth smart watch offers a better touch experience and remarkable immersion and ,more than 200 watch faces for you to choose from App"Da Fit ", you can change the dial style according to your mood to wear keeping the world on your wrist
  • 【Accurate Activity Tracker Smart Watch 】sports watch activity tracker uses sweat and dust resistant breathable silicone strap, our Fitness Tracker Smart Watch for Android iOS Phones supports 12 sports modes,walking,running,bicycle,swimming,badminton,basketball,football,rope skipping. This bluetooth calls smart watch for men women is lightweight and stylish, bringing more convenient and relaxing sports experience while also providing more protection.
  • 【Your 24 Hour Personal Daily Health Monitor】Fitness watch for women men full touch screen IP67 waterproof bluetooth smartwatch for Android iOS Phones with heart rate blood pressure monitor multifunction Andriod smart watch with steps counter sleep monitor sports fitness watch, monitoring blood oxygen saturation allows you to understand your physical condition in real time to track fitness progress ,to know your health status in time.
  • 【More Functions Bluetooth Call Smart Watch】This fitness activity smart watch full screen for iPhone and Android phones includes a bunch of new features including:The bluetooth smart watch can hang up the phone, receive and display WhatsAPP /Text /Facebook /Twitter /Messenger reminder.Support sedentary reminder, alarm clock, stopwatch, timer,weather information,music control, remote camera control compatible with Android 5.0 and above or iOS 9.0 and above smartphones.

Resolve Argo CD instance-label ownership collisions

Argo CD uses the app.kubernetes.io/instance label to determine which resources belong to an application. The Argo CD FAQ documents a failure mode in which another tool writes that same label. Argo CD can then calculate ownership incorrectly and leave the application OutOfSync even after a sync appears to succeed.

Find out whether another controller, chart, admission policy, or deployment tool overwrites the label. If the label is required by that tool, configure Argo CD to use a different application-instance label key, such as argocd.argoproj.io/instance, according to the Argo CD version deployed in your environment. Resynchronize only after the ownership key is consistent; otherwise the competing mutation will recreate the drift.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Look for a second writer

Common second writers include an autoscaler changing replica counts, an operator reconciling a custom resource, a policy engine injecting labels or sidecars, and a post-sync job editing metadata. The correct fix is to establish one owner for each field or explicitly configure Argo CD’s diff and ownership behavior. Do not “fix” the symptom by repeatedly editing the live object; reconciliation will either overwrite that edit or report it again.

Understand automated sync, retries, and reconciliation timing

Argo CD automated sync runs when an application is OutOfSync, but its documented behavior does not automatically retry a failed sync for the same commit SHA and application parameters. A retry policy can add exponential backoff. Therefore, a failed promotion may remain failed even though the application still displays the same revision and is eligible for automated sync.

  • Read the sync error and confirm whether the failure was an apply error, health failure, hook failure, or a policy rejection.
  • Check whether the desired revision or application parameters actually changed. A new retry is not guaranteed for the same SHA without a configured retry policy.
  • Inspect retry count and backoff settings before assuming the controller is stuck.
  • Check self-heal separately. Self-heal addresses live-state drift when enabled; it is not a universal repair mechanism for a failed workload or an invalid manifest.

The stable Argo CD documentation describes a default reconciliation period of 120 seconds plus up to 60 seconds of jitter, for a maximum interval of three minutes. This is a configurable default, not a timing guarantee. If a new commit has been detected but no reconciliation occurs after the configured interval, verify the controller configuration, repository access, application refresh state, and controller logs. Identify the Argo CD version before applying version-specific advice because retry and rollback behavior can change between releases.

Rank #3
New Bee Guitar Cable 10ft Electric Instrument Bass AMP Cord | 6.35mm Jack, Durable Design, Stage and Home Studio Use, Low Noise Shielded Electric Cord for Bass, Keyboard, Amp, Mixer
  • [Low Noise High Fidelity] New bee guitar cord with 22AWG oxygen-free-copper center conductors provides enhanced signal transfer, the PVC insulation material and the high-density stranded copper double shield provide noise rejection for quiet operation
  • [Ideal for Choice] New bee 1/4 straight and right angle metal connectors, ensures a secure connection in any instrument, pedal, or amp jack. L shape great fro on stage guitar stands
  • [Wide Compatible] Suitable for mixer, power amplifier, microphone, effector, decoder, equalizer, electoronic organ, electric guitar, electric bass and other electronic equipments. Widely used in the installation of smart home, karaoke speaker, professional stage audio, etc
  • [Easy Disassemble] Making incidental repairs quick and easy! Very nice, flexible cable
  • [Dont Hesite to Order] 24 hours customer services and professional technology team are standing by

Diagnose a Deployment that will not progress

Pin the rollout you are observing

Watch the Deployment with:

kubectl rollout status deployment/<name> -n <namespace>

If another rollout could start while you investigate, pin the check to the intended revision:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
kubectl rollout status deployment/<name> -n <namespace> --revision=<N>

A nonzero command status means that the observed rollout has not succeeded. It does not by itself identify the cause. Pair it with conditions and events:

kubectl describe deployment <name> -n <namespace>
kubectl get deployment <name> -n <namespace> -o yaml
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl get pods -n <namespace> -o wide

Read Deployment conditions

Look for ProgressDeadlineExceeded, FailedCreate, unavailable replicas, and a ReplicaSet that never reaches the required number of available Pods. The Deployment controller uses progressDeadlineSeconds to decide when progress is stalled. A deadline is an alert about lack of progress, not a diagnosis; inspect the underlying Pod and scheduling events.

Check the concrete failure classes

Failure class Evidence to look for Corrective direction
Insufficient quota Namespace events or ReplicaSet errors reporting quota exhaustion. Compare resource requests and limits with namespace quota; obtain an approved quota change or reduce the requested footprint.
Pod cannot be created FailedCreate, admission rejection, limit-range violation, or scheduling events. Read the complete API error and fix the rejected field, policy, limit, or scheduling constraint.
Image cannot be pulled ErrImagePull, ImagePullBackOff, authentication errors, or a missing tag/digest. Verify the image reference, registry credentials, network access, and that the same image exists in the failing cluster’s region or registry path.
Readiness never succeeds Pods run but remain Ready=False; probe failures appear in Pod events. Validate the probe path, port, scheme, startup time, dependencies, and required configuration.
Permissions or policy Forbidden API errors, service-account failures, security-policy or admission messages. Compare service accounts, RBAC, admission configuration, and namespace policies between clusters.
Limit ranges or resource settings Admission rejects missing or invalid requests and limits. Supply values required by the namespace or correct the workload’s resource specification.
Application runtime configuration Container starts and exits, crashes, or logs configuration and dependency errors. Compare ConfigMaps, Secrets, environment variables, mounted files, endpoints, and feature flags with the working cluster.

Distinguish readiness from liveness

Readiness determines whether a Pod is eligible for Service traffic. A Pod is not ready when its node is not ready, a readiness gate is false, or one or more containers are not ready; not-ready Pods are removed from Service load balancers. A failed readiness probe therefore protects traffic but can leave a rollout waiting indefinitely.

Liveness answers a different question: should the container be restarted? It should represent an unrecoverable failure, not temporary overload, slow startup, or a dependency that is expected to be unavailable briefly. An overly aggressive liveness probe can repeatedly restart a process, causing cascading failures while readiness never stabilizes. Review startup behavior, probe timeouts, failure thresholds, periods, and initial delays together.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Rank #4
BSIMB 15.6" Digital Family Calendar & Chore Chart, No Subscription Fees
  • Digital Family Calendar: BSIMB digital calendar wall touch screen puts an end to your chaotic life. Designed for busy families, the digital calendar keeps you organised and in step with your family. Assign different colours and tasks to each family member to make family scheduling easier and more intuitive.
  • Easy Setup and Auto-Sync: Plug in, connect to WLAN and auto-syn your electric calendar in minutes. Digital wall calendar compatible with Google calendar, iCloud calendar, Outlook calendar, Cozi calendar and Yahoo calendar. Add events directly on the device or through the ecalendar app. Electronic calendar stays current whether you're at work or traveling—no manual updates needed.
  • All-In-One Home Assistant: Smart calendar builds habits and encourage independence with personalised chores. Display daily meal plan to reduce the daily ‘What's for dinner?’ stress. To-do lists keep track of everything you want. Turning chores into valuable achievements with stars and rewards, making life easier and more enjoyable.
  • Manage Anytime Anywhere: BSIMB electronic calendar for family stay connected with the powerful ecalendar app. Access and manage your digital wall calendar anytime, anywhere with free features like stars & rewards, photo screensaver, chore charts, meals plans, to-do lists etc. Electronic planner paid features like Magic Import tailor advanced tools for users with special needs.
  • Stay in Touch with Loved Ones: Share photos or videos from your phone to electronic wall calendar with app, instantly update your life moment, and keep in touch with your family, especially those who are far away; you can also surprise your smart calendar digital wall by sending him a wish card; it's also great for keeping tech-illiterate elders in touch with the youngsters.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Compare a working and failing cluster systematically

Use a side-by-side worksheet rather than comparing only the Git commit. The following axes synthesize the signals exposed by Kubernetes and Argo CD; Kubernetes does not provide a single standard cross-cluster comparison API.

Axis Questions Useful evidence
Desired revision Is the same commit, image digest, chart version, and rendered overlay being promoted? Application target revision, rendered manifests, Helm values, Kustomize overlays, and image references.
Controller result Did both controllers receive and apply the same operation? Sync status, operation result, error text, retry policy, self-heal setting, and controller logs.
Resource health Are the same resources synced, available, and on the same ReplicaSet revision? Application resource tree, Deployment conditions, ReplicaSet counts, Pod phases, and rollout status.
Cluster constraints Can the failing cluster schedule and admit the same workload? Namespace quota, limit ranges, node capacity, taints, admission policies, RBAC, and service-account configuration.
Artifact access Can every cluster pull the exact image and reach required dependencies? Image-pull events, registry credentials, network policy, DNS, endpoints, and region-specific registry access.
Traffic eligibility Are Pods ready and included in Service endpoints? Readiness events, readiness gates, Service endpoint slices, and probe configuration.

When the rendered target differs, fix the promotion inputs. When targets match but sync results differ, investigate controller access, permissions, admission, or repository state. When both are synced but health differs, focus on workload and environment evidence rather than triggering another sync.

Contain a failed canary without destroying evidence

If monitoring identifies a canary problem, protect stable capacity while preserving the failed version for inspection.

  1. Stop canary traffic or promotion. Follow the deployment system’s documented pause mechanism so no additional stage advances while evidence is collected.
  2. Scale the canary Deployment to zero.
    kubectl scale deployment/<canary-name> --replicas=0 -n <namespace>

    Kubernetes’ canary tutorial notes that this removes canary Pods while preserving the Deployment configuration for inspection.

  3. Restore stable capacity if required. Scale the stable Deployment back to its intended replica count if the canary reduced available capacity or if traffic has shifted unexpectedly.
  4. Collect evidence. Save the canary manifest, ReplicaSet revision, Pod logs, events, probe failures, image digest, and rendered configuration before deleting anything.
  5. Correct the declared source. Fix the Git manifest, values, image, probe, policy, or environment configuration that caused the failure, then let the promotion workflow reconcile it.
  6. Promote again only after health gates pass. Verify readiness, rollout completion, and application-level health before restoring canary traffic.

Rollback safely when GitOps owns the application

Direct Kubernetes-managed Deployment

For a Deployment managed directly through Kubernetes, kubectl rollout undo can restore an earlier Pod-template revision:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
kubectl rollout history deployment/<name> -n <namespace>
kubectl rollout undo deployment/<name> -n <namespace> --to-revision=<N>
kubectl rollout status deployment/<name> -n <namespace>

Revisions are created when the Pod template changes. Scaling-only changes do not create a new revision, and rollback restores the Pod template rather than every historical aspect of the application. Confirm the resulting image, configuration references, and replica behavior after the undo.

Argo CD automated sync

Argo CD documents that rollback cannot be performed against an application while automated sync is enabled. In that situation, an imperative live-cluster rollback is temporary: reconciliation can overwrite it with the declared state. Revert or correct the desired configuration in Git through the normal review and deployment workflow, then confirm that Argo CD reconciles the new revision and that the Deployment becomes healthy.

If an emergency imperative action is unavoidable, record it, protect stable capacity, and follow it immediately with a source-of-truth change. Otherwise the cluster and Git will diverge again, producing the same drift signal during the next reconciliation.

Recovery decision tree

  • OutOfSync and rendered target is wrong: correct the manifest or promotion inputs, commit the change, and verify the new revision.
  • OutOfSync and live state is being mutated: identify the second writer, resolve label or field ownership, then resynchronize.
  • Sync failed before resources changed: fix the reported API, permission, admission, repository, or artifact error; check retry behavior before waiting for another attempt.
  • Synced but Deployment is stalled: stop triggering syncs and inspect conditions, ReplicaSet, Pods, events, quota, images, permissions, configuration, and probes.
  • Canary is unhealthy: scale it to zero, preserve its evidence, restore stable replicas if needed, and correct the declared source before another promotion.
  • Rollback is required under automated sync: change Git rather than relying on a live-only undo, then verify reconciliation and workload health.

Version and timing cautions

Argo CD automated-sync defaults, retry behavior, rollback restrictions, and resource-health semantics are version-sensitive. The stable documentation describes the 120-second reconciliation period plus up to 60 seconds of jitter, but operators can change that setting. Identify the Argo CD release running in the affected environment before relying on a UI label, controller default, or rollback behavior.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Kubernetes command flags and Deployment behavior should likewise be checked against the cluster’s supported Kubernetes release. The commands above are documented procedures; they should be run with the appropriate context and permissions, and their output should be captured as incident evidence.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Argo CDcanary releasescluster troubleshootingdeployment rolloutsGitOpsKubernetesSRE

Leave a Reply

Your email address will not be published. Required fields are marked *

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Recommended PC Tool
Recommended PC Tool
Windows Errors? Fix Them Before They SpreadFree repair scan
Crashes, No Sound, or Screen Glitches?Free driver scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.