Halloween DealsAmazon USHalloween week deals: useful picks before month-endAmazon US: current deals, useful picks and tech finds.Check DealsWindows FixRecommendedWindows errors stealing your time? Find the fix fastScan stability, cleanup and performance issues.Fix NowHalloween DealsAmazon USOctober picks are moving - compare before checkoutAmazon US: current deals, useful picks and tech finds.See Picks×
FREEDOM251

Freedom251 › Blog › Amazon ECS

What ECS GPU Auto-Repair Does—and What It Does Not Fix

By Freedom251 Team · · 11 min read

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Amazon ECS GPU auto-repair replaces an impaired ECS Managed Instance when NVIDIA Data Center GPU Manager (DCGM) reports one of the critical GPU failures that ECS monitors. It does not repair a GPU in place, restart broken application code, correct a container or CUDA configuration, or diagnose every possible graphics-processing problem. The sequence is infrastructure replacement: ECS stops new task placement on the instance, brings up replacement capacity, lets existing tasks stop within their configured timeout, and then terminates the impaired instance.

What ECS GPU auto-repair actually is

ECS GPU auto-repair is a health-triggered replacement mechanism for Amazon ECS Managed Instances that use NVIDIA GPU hardware. ECS relies on NVIDIA Data Center GPU Manager (DCGM) to report critical GPU health failures. When a monitored failure is detected, ECS marks the container instance IMPAIRED. With automatic repair actions enabled, ECS replaces that managed instance rather than attempting an in-place component repair.

As an Amazon Associate I earn from qualifying purchases.

That distinction matters operationally. The feature protects the capacity-provider fleet from a failed GPU instance, but it is not a general-purpose recovery system for workloads running on the instance. A task can still fail because of application code, a bad model, an incompatible CUDA or driver setup, an invalid device request, insufficient memory, or an external dependency. None of those causes is automatically corrected merely because the instance is replaced.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

AWS describes the feature as monitoring NVIDIA GPU health on GPU-enabled ECS Managed Instances. The documented behavior is bounded by the DCGM signals and NVIDIA Xid errors that ECS recognizes. If a problem does not produce one of those monitored critical signals, ECS GPU auto-repair is not documented as a response to it.

#1 Best Overall
Sale
TOZO A1 Wireless Earbuds Bluetooth 5.3 Light Weight in Ear Headphones | IPX5 Waterproof,2 Mics for AI Calls,Immersive Premium Sound Bass Headset with Charging Case,32 EQ via App Black
  • [Ultra-Lightweight Ear Buds Designed for Small Ears] Each earbud weighs only 3.7g and features a compact, ergonomic in-ear design made especially for small ears. Secure, low-profile, and comfortable for workouts, all-day wear or overnight sleeping.The ultra-slim profile sits flush in your ear with no hard edges poking or rubbing, so you can even sleep on your side without discomfort.
  • [Immersive Stereo Sound with TOZO OrigX Technology] TOZO OrigX tuning delivers clear vocals, balanced mids, and natural stereo sound for music, podcasts, and videos.
  • [Long Battery Life for Daily Use] Get up to 7 hours of playtime on a single charge, with up to 32 hours total using the charging case—ideal for workdays, commuting, and extended listening sessions.
  • [Bluetooth 5.3 & Stable Connection] Bluetooth 5.3 provides fast pairing, stable wireless performance, and reduced dropouts as you move around home or office.
  • [Deep Bass with Clear Vocals] High-performance drivers produce punchy bass while keeping vocals clean and detailed for everyday listening.

What events trigger replacement

The ECS GPU auto-repair guide lists the following NVIDIA Xid codes as monitored. If ECS detects one of these failures through DCGM, it marks the instance impaired; when repair actions are enabled, the instance enters the replacement workflow.

Xid code GPU condition reported by NVIDIA
46 GPU stopped processing
48 Double-bit ECC error
54 Auxiliary power connector not connected
62 Internal micro-controller halt
64 GPU memory remapping failure
74 NVLink error
79 GPU fell off the bus
95 Uncontained memory error
109 Context switch timeout
110 GPU disappeared from the bus
136 GPU memory page retirement limit exceeded
140 Unrecoverable ECC error
142 GPU memory page retired because of an uncorrectable error
143 GPU memory page retired after a correctable-error threshold was reached
151 GPU-to-CPU interconnect error
155 GPU NVLink flit CRC error
156 GPU NVLink lane error
158 GPU InfoROM corrupted

The list is a trigger list, not a promise that every GPU symptom maps to one of these codes. For example, an application reporting slow inference, a task failing to initialize its device, or a container losing access to a library does not by itself prove that a monitored Xid occurred. Check the ECS health reason and the underlying task and host logs before treating the event as an auto-repair case.

How the replacement workflow proceeds

Replacement uses a start-before-stop sequence designed to preserve capacity while the impaired host is drained. The exact duration depends on provisioning and task shutdown behavior; AWS publishes concurrency limits, not a guaranteed repair time.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  1. ECS detects the critical GPU condition. DCGM reports a monitored failure and ECS changes the container instance health state to impaired.
  2. The impaired instance is set to DRAINING. ECS stops placing new tasks on that instance. Tasks already running there remain until they stop or their configured stop behavior takes effect.
  3. ECS provisions replacement capacity. The managed capacity provider starts a replacement instance. This is a new instance, not a repaired GPU component inside the old one.
  4. Existing tasks receive a graceful stop window. ECS allows tasks on the draining instance to stop according to their task stop timeout. Applications that handle termination signals can use this window to checkpoint, flush work, or close connections.
  5. The impaired instance is terminated after draining. Once the drain period and task shutdown process complete, ECS removes the failed instance.

Because tasks are allowed to stop, auto-repair should not be described as guaranteeing uninterrupted execution. A service with more than one healthy task and suitable placement capacity may continue serving traffic while one task is replaced, but that outcome depends on the service’s desired count, deployment and load-balancing settings, available capacity, and the application’s shutdown behavior. The ECS documentation supports graceful stopping and instance replacement; it does not promise task continuity for every workload.

Why the task stop timeout matters

The task stop timeout controls how long ECS gives a running task to exit gracefully during draining. A short timeout can cause work to be cut off before checkpoints or cleanup finish. A long timeout delays termination of the impaired instance and can keep resources occupied while replacement capacity is already being prepared. Set the timeout to match the workload’s realistic shutdown and checkpoint needs, then test that behavior under a controlled drain.

What auto-repair fixes

Auto-repair addresses one infrastructure condition: a GPU-enabled managed instance that ECS has marked impaired because DCGM reported a monitored critical GPU failure. Its corrective action is to remove that instance from placement and replace it with new managed capacity.

Rank #2
Sale
Smart Watch for Men Women Compatible with iPhone Samsung Android Phone 1.69 inch Full Touch Screen IP68 Waterproof Bluetooh Fitness Tracker Heart Rate/Sleep Monitor (Black)
  • 【IP68 Waterproof Multifunctinal Smart Watch】Torjalph fitness activity tracker smart watch is IP68 waterproof technology ,you can wear it when swimming,handwashing,it's rain-proof which means that you perform multiple exercise modes outside without worrying about sweat or rain corroding your fitness smart watch,and this watch equipped with a 230 mAh large capacity battery magnetic charging ,and this sports activity tracker watch can be used 5-7 days If fully charged.
  • 【Larger 1.83'' Full Screen Watch & Multi-dial Selection】1.83 inch full HD color touch screen smart watch for men / women provides excellent 240*284 resolution image quality ,This Bluetooth smart watch offers a better touch experience and remarkable immersion and ,more than 200 watch faces for you to choose from App"Da Fit ", you can change the dial style according to your mood to wear keeping the world on your wrist
  • 【Accurate Activity Tracker Smart Watch 】sports watch activity tracker uses sweat and dust resistant breathable silicone strap, our Fitness Tracker Smart Watch for Android iOS Phones supports 12 sports modes,walking,running,bicycle,swimming,badminton,basketball,football,rope skipping. This bluetooth calls smart watch for men women is lightweight and stylish, bringing more convenient and relaxing sports experience while also providing more protection.
  • 【Your 24 Hour Personal Daily Health Monitor】Fitness watch for women men full touch screen IP67 waterproof bluetooth smartwatch for Android iOS Phones with heart rate blood pressure monitor multifunction Andriod smart watch with steps counter sleep monitor sports fitness watch, monitoring blood oxygen saturation allows you to understand your physical condition in real time to track fitness progress ,to know your health status in time.
  • 【More Functions Bluetooth Call Smart Watch】This fitness activity smart watch full screen for iPhone and Android phones includes a bunch of new features including:The bluetooth smart watch can hang up the phone, receive and display WhatsAPP /Text /Facebook /Twitter /Messenger reminder.Support sedentary reminder, alarm clock, stopwatch, timer,weather information,music control, remote camera control compatible with Android 5.0 and above or iOS 9.0 and above smartphones.
  • It removes the impaired instance from new task placement by using the DRAINING state.
  • It creates replacement capacity through the ECS managed capacity provider.
  • It gives existing tasks a configured opportunity to stop gracefully.
  • It terminates the impaired instance after the drain process.
  • It keeps monitoring health even when replacement actions are disabled, provided the instance remains under ECS health monitoring.

The result is a new host that can supply a functioning GPU, assuming replacement provisioning succeeds and the new instance itself is healthy. ECS does not claim that the original GPU is repaired, that data on the old host is recovered, or that a replacement instance will correct a software defect.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

What auto-repair does not fix

The documented trigger and action are deliberately narrow. Treat the following as operator-owned problems unless a separate ECS or application mechanism addresses them.

Application and model failures

A segmentation fault, deadlock, out-of-memory condition in the application, malformed model, bad input, or failed external API call is not established as a GPU auto-repair trigger. Replacing the host may cause the task to be relaunched, but it does not change the application binary, model, input, or dependency that caused the failure.

Container, driver, and CUDA configuration errors

An image that lacks the required libraries, a mismatched CUDA and driver combination, an incorrect GPU resource request, or a permissions problem can prevent a task from using its GPU without producing one of the monitored critical Xids. Investigate the task definition, image, runtime, driver compatibility, and device discovery separately.

Unmonitored or ambiguous hardware symptoms

The published Xid list is the documented scope. A temperature alert, throttling event, intermittent performance loss, power problem that does not surface as a listed code, or another hardware symptom is not automatically covered unless ECS receives a monitored DCGM failure. Use host and GPU telemetry to determine whether the issue is within the auto-repair boundary.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Data recovery and state restoration

Instance replacement does not restore data stored only on the failed host. Persist checkpoints, model artifacts, queues, and other durable state in storage designed for failure recovery. A draining task can perform a final checkpoint only if the application supports it and the stop timeout is long enough.

Rank #3
New Bee Guitar Cable 10ft Electric Instrument Bass AMP Cord | 6.35mm Jack, Durable Design, Stage and Home Studio Use, Low Noise Shielded Electric Cord for Bass, Keyboard, Amp, Mixer
  • [Low Noise High Fidelity] New bee guitar cord with 22AWG oxygen-free-copper center conductors provides enhanced signal transfer, the PVC insulation material and the high-density stranded copper double shield provide noise rejection for quiet operation
  • [Ideal for Choice] New bee 1/4 straight and right angle metal connectors, ensures a secure connection in any instrument, pedal, or amp jack. L shape great fro on stage guitar stands
  • [Wide Compatible] Suitable for mixer, power amplifier, microphone, effector, decoder, equalizer, electoronic organ, electric guitar, electric bass and other electronic equipments. Widely used in the installation of smart home, karaoke speaker, professional stage audio, etc
  • [Easy Disassemble] Making incidental repairs quick and easy! Very nice, flexible cable
  • [Dont Hesite to Order] 24 hours customer services and professional technology team are standing by

Capacity shortages and placement constraints

Auto-repair requests replacement capacity; it does not guarantee that a matching GPU instance is immediately available. Quotas, subnet capacity, placement constraints, launch-template errors, or a full capacity provider can prevent replacement from becoming usable. Monitor the replacement attempt and resolve those infrastructure conditions independently.

Default behavior and configuration choices

The current AWS GPU auto-repair guide says the feature is enabled by default for Amazon ECS Managed Instances. You can change the action state in the capacity provider’s autoRepairConfiguration when creating or updating the capacity provider.

Setting Replacement behavior Health monitoring Important interaction
ENABLED (the documented default) ECS replaces instances reported IMPAIRED after draining. Continues. Managed Daemons auto-repair remains available according to its own conditions.
DISABLED ECS does not automatically replace impaired instances. Continues; ECS still reports health state and reasons. Disabling GPU auto-repair also disables Amazon ECS Managed Daemons auto-repair.

To disable automatic replacement, set actionsStatus to DISABLED inside autoRepairConfiguration while creating or updating the capacity provider. The API reference defines enabled actions as replacing instances reported IMPAIRED and disabled actions as monitoring without replacement. Disabling the action is therefore not the same as disabling health checks.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Choose the disabled state only when you have an explicit operator workflow for impaired hosts. For example, a maintenance process might need to inspect a host before replacement, coordinate a stateful checkpoint, or pause automated changes during an incident. During that period, the instance can remain impaired and continue to receive no new placements if it is draining, while your team handles the replacement decision.

How to see the health state and Xid reason

DescribeContainerInstances exposes the health information ECS aggregates for a container instance. On ECS Managed Instances, individual checks include ACCELERATED_COMPUTE, the check associated with GPU health. The overall container-instance health state is derived from the individual checks.

For a GPU impairment, the accelerated-compute check’s status reason contains the NVIDIA code in the form XID_<number>. That reason is the fastest way to distinguish a documented GPU trigger from a generic task failure. Record the code, instance identifier, task placements, and event time before the instance is terminated so that incident analysis does not depend on the old host remaining available.

Rank #4
BSIMB 15.6" Digital Family Calendar & Chore Chart, No Subscription Fees
  • Digital Family Calendar: BSIMB digital calendar wall touch screen puts an end to your chaotic life. Designed for busy families, the digital calendar keeps you organised and in step with your family. Assign different colours and tasks to each family member to make family scheduling easier and more intuitive.
  • Easy Setup and Auto-Sync: Plug in, connect to WLAN and auto-syn your electric calendar in minutes. Digital wall calendar compatible with Google calendar, iCloud calendar, Outlook calendar, Cozi calendar and Yahoo calendar. Add events directly on the device or through the ecalendar app. Electronic calendar stays current whether you're at work or traveling—no manual updates needed.
  • All-In-One Home Assistant: Smart calendar builds habits and encourage independence with personalised chores. Display daily meal plan to reduce the daily ‘What's for dinner?’ stress. To-do lists keep track of everything you want. Turning chores into valuable achievements with stars and rewards, making life easier and more enjoyable.
  • Manage Anytime Anywhere: BSIMB electronic calendar for family stay connected with the powerful ecalendar app. Access and manage your digital wall calendar anytime, anywhere with free features like stars & rewards, photo screensaver, chore charts, meals plans, to-do lists etc. Electronic planner paid features like Magic Import tailor advanced tools for users with special needs.
  • Stay in Touch with Loved Ones: Share photos or videos from your phone to electronic wall calendar with app, instantly update your life moment, and keep in touch with your family, especially those who are far away; you can also surprise your smart calendar digital wall by sending him a wish card; it's also great for keeping tech-illiterate elders in touch with the youngsters.

ECS container-instance health change events provide an event-driven way to observe these transitions. Use them with your event routing and incident tooling to alert when an instance becomes impaired, enters draining, or changes back to a healthy state. The event tells you what ECS observed; it does not replace application logs, driver logs, or task-level diagnostics.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

GPU metrics: basic versus enhanced Container Insights

Health state and Xid reasons tell you that ECS has identified an impairment. They do not provide the full performance history of every GPU. For that detail, AWS supports DCGM GPU metrics through Container Insights with enhanced observability on supported GPU-enabled ECS Managed Instances.

Observability option GPU metric coverage Resource levels Cost note
Basic Container Insights Does not collect GPU metrics. General Container Insights data only. GPU metrics are not included.
Container Insights with enhanced observability Collects DCGM GPU metrics on supported GPU-enabled ECS Managed Instances. Container, task, and instance levels. AWS notes that enhanced Container Insights incurs additional cost.

Use enhanced metrics when you need trend and utilization context around an Xid, such as whether memory pressure or an interconnect problem preceded the impairment. Do not confuse metric visibility with repair coverage: enhanced observability improves diagnosis, but it does not expand the documented set of auto-repair triggers.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Repair rate limits and capacity planning

ECS rate-limits replacements to reduce the risk of a cascading failure. In one capacity provider, no more than 20% of instances can be draining at the same time. If the capacity provider has fewer than nine instances, at most one instance can drain at a time.

Capacity-provider size Maximum concurrent draining instances
Fewer than 9 instances 1
9 or more instances No more than 20% of instances

These are concurrency limits, not a service-level objective for how quickly a replacement will finish. A small GPU fleet can therefore repair serially, while a larger fleet can repair several instances but still only within the 20% ceiling. Plan desired task counts and spare capacity so that losing the allowed number of instances does not leave a service below its availability target.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Managed Daemons: the configuration interaction to understand

Disabling GPU auto-repair also disables Amazon ECS Managed Daemons auto-repair. That coupling can be easy to miss when an operator changes only the GPU setting.

Managed Daemons can drain and replace an instance when a critical daemon task stops or becomes unhealthy. A daemon health check is optional. If no health check is configured, ECS can detect a daemon failure only when the task stops; an unhealthy-but-running daemon may not trigger that mechanism. When GPU auto-repair actions are disabled, do not assume daemon auto-repair will continue independently. Review daemon requirements before changing actionsStatus.

A practical incident workflow

1. Confirm that ECS, not the application, declared impairment

Inspect the container instance through DescribeContainerInstances and look for the ACCELERATED_COMPUTE check. Capture its status reason, especially an XID_<number> value, and note the instance health state.

2. Classify the event against the monitored list

Compare the reported Xid with the documented codes. A listed code explains why ECS can initiate replacement. A missing code means you should investigate other health checks, task events, driver logs, and application telemetry rather than assuming GPU auto-repair applies.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

3. Check draining and replacement progress

Verify that the impaired instance is no longer receiving new placements and that replacement capacity is being provisioned. Check capacity-provider events for launch, quota, subnet, or placement failures. If replacement cannot launch, resolve that capacity problem; auto-repair cannot manufacture unavailable GPU capacity.

4. Protect in-flight work

Use the task stop timeout and application signal handling to checkpoint or flush work. Confirm that service deployments maintain enough healthy tasks while the impaired host drains. Batch workloads should make checkpoints durable before termination rather than relying on the old instance’s local storage.

5. Preserve evidence before termination

Save the Xid reason, ECS event, instance and task identifiers, enhanced Container Insights data if enabled, and relevant application and driver logs. The old instance is terminated after draining, so evidence left only on its local filesystem may disappear.

6. Investigate recurring failures separately

Repeated replacement of otherwise identical instances suggests a fleet, image, driver, power, interconnect, or placement issue that deserves a broader investigation. Auto-repair limits impact by replacing failed capacity; it does not identify the systemic cause or prevent recurrence.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Common misconceptions and the precise correction

  • “Auto-repair fixes my GPU.” It replaces the managed instance; the documentation does not describe an in-place GPU repair.
  • “Any GPU error will trigger it.” ECS monitors the published DCGM/Xid conditions, not every possible GPU symptom.
  • “Tasks will keep running without interruption.” Tasks on the draining instance are allowed to stop; continuity depends on service redundancy, replacement capacity, and application behavior.
  • “Turning it off stops monitoring.” Setting actionsStatus to DISABLED stops automatic replacement while health monitoring continues.
  • “Enhanced metrics cause repairs.” Enhanced Container Insights supplies DCGM metrics for diagnosis; repair decisions still use the ECS health mechanism.
  • “A disabled GPU setting affects only GPUs.” AWS states that disabling GPU auto-repair also disables Managed Daemons auto-repair.

Decision checklist for operators

  • Is the capacity provider using Amazon ECS Managed Instances with NVIDIA GPU hardware?
  • Is autoRepairConfiguration.actionsStatus enabled or intentionally disabled?
  • Do alerts capture ACCELERATED_COMPUTE health changes and the XID_<number> reason?
  • Are task stop timeouts long enough for checkpointing, but short enough to release impaired capacity promptly?
  • Can the capacity provider launch a replacement GPU instance within its quotas, subnets, and placement constraints?
  • Does the service have enough healthy tasks to tolerate one or more draining instances under the 20% or single-instance limit?
  • Is enhanced Container Insights enabled when container-, task-, and instance-level DCGM metrics are required, and is its additional cost accepted?
  • Will disabling GPU auto-repair unintentionally remove the required Managed Daemons auto-repair behavior?
  • Are durable checkpoints and logs stored outside the instance that ECS will terminate?

The useful mental model is simple: ECS GPU auto-repair is a monitored-instance replacement path. DCGM reports a defined critical GPU failure, ECS marks the instance impaired, drains it, starts replacement capacity, honors task stop timeouts, and terminates the old host. Everything outside that health signal and replacement sequence—software defects, workload design, data durability, capacity shortages, and root-cause analysis—remains the operator’s responsibility.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Amazon ECSCloud Infrastructurecontainer orchestrationDCGMGPUNVIDIA

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Recommended PC Tool
Recommended PC Tool
Crashes, No Sound, or Screen Glitches?Free driver scan
Windows Errors? Fix Them Before They SpreadFree repair scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.