Halloween DealsAmazon USHalloween week deals: useful picks before month-endAmazon US: current deals, useful picks and tech finds.Check DealsWindows FixRecommendedWindows errors stealing your time? Find the fix fastScan stability, cleanup and performance issues.Fix NowHalloween DealsAmazon USOctober picks are moving - compare before checkoutAmazon US: current deals, useful picks and tech finds.See Picks×
FREEDOM251

Freedom251 › Blog › Amazon ECS

How to Configure Amazon ECS GPU Instance Health Checks and Automatic Recovery

By Freedom251 Team · · 10 min read

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Use Amazon ECS Managed Instances when you need ECS to detect a failed NVIDIA GPU and replace the host automatically. The managed-instance health system uses NVIDIA Data Center GPU Manager (DCGM) checks, reports an impaired container instance when a critical GPU fault is detected, and—when the capacity provider’s repair action is enabled—drains that instance and provisions a replacement.

This workflow is specific to Amazon ECS Managed Instances. It should not be presented as a built-in equivalent for every ECS cluster that uses ordinary EC2 instances or an Auto Scaling group. The procedure below shows how to create or verify the managed capacity provider, inspect GPU health, understand the replacement sequence, and distinguish host failures from unhealthy application containers.

As an Amazon Associate I earn from qualifying purchases.

What ECS GPU instance health checks actually cover

Amazon ECS monitors NVIDIA GPU health on Amazon ECS Managed Instances that use GPU hardware. The check is an instance-level signal: it evaluates the GPU and the managed host/runtime rather than a command inside one container. ECS uses NVIDIA DCGM to collect the GPU health signal. A critical failure can change the container instance to IMPAIRED; the reported reason may contain an NVIDIA XID identifier such as XID_79 or another XID_<number> value.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

The distinction matters operationally. A GPU can fail while every application process in a container still appears healthy, or a container can fail while the physical GPU is fine. ECS treats those as different events and gives them different recovery paths.

#1 Best Overall
Sale
TOZO A1 Wireless Earbuds Bluetooth 5.3 Light Weight in Ear Headphones | IPX5 Waterproof,2 Mics for AI Calls,Immersive Premium Sound Bass Headset with Charging Case,32 EQ via App Black
  • [Ultra-Lightweight Ear Buds Designed for Small Ears] Each earbud weighs only 3.7g and features a compact, ergonomic in-ear design made especially for small ears. Secure, low-profile, and comfortable for workouts, all-day wear or overnight sleeping.The ultra-slim profile sits flush in your ear with no hard edges poking or rubbing, so you can even sleep on your side without discomfort.
  • [Immersive Stereo Sound with TOZO OrigX Technology] TOZO OrigX tuning delivers clear vocals, balanced mids, and natural stereo sound for music, podcasts, and videos.
  • [Long Battery Life for Daily Use] Get up to 7 hours of playtime on a single charge, with up to 32 hours total using the charging case—ideal for workdays, commuting, and extended listening sessions.
  • [Bluetooth 5.3 & Stable Connection] Bluetooth 5.3 provides fast pairing, stable wireless performance, and reduced dropouts as you move around home or office.
  • [Deep Bass with Clear Vocals] High-performance drivers produce punchy bass while keeping vocals clean and detailed for everyday listening.
  • GPU or host impairment: Managed Instances health can trigger instance draining and replacement when automatic repair is enabled.
  • Application failure: A task-definition health-check command can mark one task unhealthy, allowing the service scheduler to stop it and launch another task.
  • Telemetry: Container Insights enhanced observability can expose DCGM metrics for GPU-enabled Managed Instances; basic Container Insights does not collect those GPU metrics.

Prerequisites before enabling GPU auto repair

Automatic replacement starts with the compute model, not with a task-definition setting. Confirm these conditions before changing a provider:

  • Your cluster uses an Amazon ECS Managed Instances capacity provider.
  • The provider has the infrastructure role and instance profile required for Managed Instances.
  • The provider is associated with the intended cluster and network configuration, including subnets, security groups, and connectivity needed by ECS.
  • Its instance requirements select GPU capacity, either by specifying an accelerator type and count or by selecting suitable GPU instance types.
  • Container instances run an ECS agent version that supports automated container-instance checks. AWS documents version 1.57.0 or later for those checks.
  • Your service or task uses the provider’s capacity-provider strategy, so replacement capacity is eligible to run the workload.

In the ECS console, create a capacity provider using the Managed Instances option, choose the cluster and networking, configure the required roles and instance requirements, and include the GPU accelerator requirement. The same provider can be created or changed through the ECS capacity-provider APIs. Do not substitute an Auto Scaling group-backed provider for a Managed Instances provider and assume that the Managed Instances GPU repair behavior follows automatically.

Enable automatic repair on the Managed Instances provider

The repair switch belongs to the capacity provider’s managedInstancesProvider.autoRepairConfiguration object. Its actionsStatus value controls what ECS does after health monitoring declares an instance impaired:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
actionsStatus Monitoring behavior Repair behavior
ENABLED ECS continues accelerated-compute and other managed instance checks. ECS can drain and replace an impaired instance automatically.
DISABLED ECS still reports health results and impairment details. No automatic repair action is taken by this feature; an operator or another recovery system must respond.

AWS’s GPU auto-repair documentation says automatic repair is enabled by default for Managed Instances. Treat that as a documented default, not as proof of the current state of your provider. Inspect the actual provider configuration after creation and after any infrastructure change.

Illustrative provider configuration

The following fragment shows the relevant field. The values are an example of the object shape, not a complete tested create command:

Rank #2
Sale
Smart Watch for Men Women Compatible with iPhone Samsung Android Phone 1.69 inch Full Touch Screen IP68 Waterproof Bluetooh Fitness Tracker Heart Rate/Sleep Monitor (Black)
  • 【IP68 Waterproof Multifunctinal Smart Watch】Torjalph fitness activity tracker smart watch is IP68 waterproof technology ,you can wear it when swimming,handwashing,it's rain-proof which means that you perform multiple exercise modes outside without worrying about sweat or rain corroding your fitness smart watch,and this watch equipped with a 230 mAh large capacity battery magnetic charging ,and this sports activity tracker watch can be used 5-7 days If fully charged.
  • 【Larger 1.83'' Full Screen Watch & Multi-dial Selection】1.83 inch full HD color touch screen smart watch for men / women provides excellent 240*284 resolution image quality ,This Bluetooth smart watch offers a better touch experience and remarkable immersion and ,more than 200 watch faces for you to choose from App"Da Fit ", you can change the dial style according to your mood to wear keeping the world on your wrist
  • 【Accurate Activity Tracker Smart Watch 】sports watch activity tracker uses sweat and dust resistant breathable silicone strap, our Fitness Tracker Smart Watch for Android iOS Phones supports 12 sports modes,walking,running,bicycle,swimming,badminton,basketball,football,rope skipping. This bluetooth calls smart watch for men women is lightweight and stylish, bringing more convenient and relaxing sports experience while also providing more protection.
  • 【Your 24 Hour Personal Daily Health Monitor】Fitness watch for women men full touch screen IP67 waterproof bluetooth smartwatch for Android iOS Phones with heart rate blood pressure monitor multifunction Andriod smart watch with steps counter sleep monitor sports fitness watch, monitoring blood oxygen saturation allows you to understand your physical condition in real time to track fitness progress ,to know your health status in time.
  • 【More Functions Bluetooth Call Smart Watch】This fitness activity smart watch full screen for iPhone and Android phones includes a bunch of new features including:The bluetooth smart watch can hang up the phone, receive and display WhatsAPP /Text /Facebook /Twitter /Messenger reminder.Support sedentary reminder, alarm clock, stopwatch, timer,weather information,music control, remote camera control compatible with Android 5.0 and above or iOS 9.0 and above smartphones.
{
  "managedInstancesProvider": {
    "autoRepairConfiguration": {
      "actionsStatus": "ENABLED"
    }
  }
}

When creating a provider, supply the remaining required fields for roles, networking, and instance requirements as described in the ECS Managed Instances capacity-provider API. For an existing provider, use the provider update operation to set or change autoRepairConfiguration.actionsStatus, then read the provider back and verify the resulting value. Keep that verification in your deployment or infrastructure-change checklist; a later update can otherwise leave monitoring enabled while repair actions are disabled.

How ECS detects and records a GPU failure

ECS runs several container-instance checks. The overall result is the most severe status among the checks currently available. The documented overall values are:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Status Meaning for operations
OK Checks currently report a healthy instance.
IMPAIRED At least one critical check has failed. A GPU-related failure can place the instance here.
INSUFFICIENT_DATA ECS cannot obtain enough information to determine health.
INITIALIZING Checks are still starting and have not produced a final health result.

Use DescribeContainerInstances with the CONTAINER_INSTANCE_HEALTH include value to retrieve the detailed result. The AWS CLI request has this form:

aws ecs describe-container-instances 
  --cluster <cluster-name-or-arn> 
  --container-instances <container-instance-arn-or-id> 
  --include CONTAINER_INSTANCE_HEALTH

Replace the angle-bracket values with your cluster and container-instance identifiers. In the response, inspect healthStatus.overallStatus and each entry in healthStatus.details. For a GPU incident, also inspect the container instance’s statusReason for an NVIDIA XID code. The XID is a diagnostic clue from the NVIDIA driver/GPU stack, not a promise that every XID has the same physical cause; correlate it with the health-change event, task placement, draining activity, and GPU telemetry.

What happens after an impaired instance is found

With actionsStatus set to ENABLED, ECS uses a start-before-stop replacement sequence. It does not simply reboot the failed host and keep tasks running on it.

Rank #3
New Bee Guitar Cable 10ft Electric Instrument Bass AMP Cord | 6.35mm Jack, Durable Design, Stage and Home Studio Use, Low Noise Shielded Electric Cord for Bass, Keyboard, Amp, Mixer
  • [Low Noise High Fidelity] New bee guitar cord with 22AWG oxygen-free-copper center conductors provides enhanced signal transfer, the PVC insulation material and the high-density stranded copper double shield provide noise rejection for quiet operation
  • [Ideal for Choice] New bee 1/4 straight and right angle metal connectors, ensures a secure connection in any instrument, pedal, or amp jack. L shape great fro on stage guitar stands
  • [Wide Compatible] Suitable for mixer, power amplifier, microphone, effector, decoder, equalizer, electoronic organ, electric guitar, electric bass and other electronic equipments. Widely used in the installation of smart home, karaoke speaker, professional stage audio, etc
  • [Easy Disassemble] Making incidental repairs quick and easy! Very nice, flexible cable
  • [Dont Hesite to Order] 24 hours customer services and professional technology team are standing by
  1. Drain the impaired instance. ECS changes the container instance to DRAINING, preventing new tasks from being placed there.
  2. Provision replacement capacity. ECS requests a replacement Managed Instance that satisfies the capacity provider’s GPU and other instance requirements.
  3. Stop existing tasks gracefully. Tasks on the draining host are allowed to stop according to their configured stop timeout. Applications should handle termination signals and checkpoint or flush work where appropriate.
  4. Terminate the impaired instance. After draining, ECS terminates the failed instance rather than returning it to service.

Replacement is deliberately capacity-aware. ECS limits concurrent draining to no more than 20 percent of a capacity provider’s instances. For a provider with fewer than nine instances, no more than one instance drains at a time. A small GPU fleet can therefore recover serially, and a larger fleet can still take time if replacement capacity is constrained by quotas, subnets, placement, or regional availability.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Because tasks leave the failed host, this process is not uninterrupted execution on the same machine. Use more than one healthy GPU instance, suitable service deployment settings, and application-level checkpointing when the workload cannot tolerate a task restart.

Health checks at the container and instance layers

Do not use a task health check as a substitute for GPU host monitoring. The two mechanisms evaluate different things and produce different actions.

Layer What it evaluates Typical response Correct description
ECS task or container health A health-check command configured in the task definition and executed inside the container. If a service task becomes unhealthy, the ECS service scheduler stops it and launches a replacement task according to service deployment rules. Application or container health.
ECS Managed Instances container-instance health Host and runtime checks, ECS agent connectivity, accelerated-compute health, and supported managed-daemon checks. When the instance is impaired and automatic repair is enabled, ECS drains and replaces the instance. Instance or GPU hardware health.

A container health check can confirm that a CUDA application responds to a command, but it cannot guarantee that the host GPU is free of driver-level faults. Conversely, a DCGM failure can impair the host even when the application’s last health command succeeded. Configure both when you need protection at both layers.

Observe GPU health and repair events

For an operational view, combine ECS health details, ECS events, and CloudWatch metrics:

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Rank #4
BSIMB 15.6" Digital Family Calendar & Chore Chart, No Subscription Fees
  • Digital Family Calendar: BSIMB digital calendar wall touch screen puts an end to your chaotic life. Designed for busy families, the digital calendar keeps you organised and in step with your family. Assign different colours and tasks to each family member to make family scheduling easier and more intuitive.
  • Easy Setup and Auto-Sync: Plug in, connect to WLAN and auto-syn your electric calendar in minutes. Digital wall calendar compatible with Google calendar, iCloud calendar, Outlook calendar, Cozi calendar and Yahoo calendar. Add events directly on the device or through the ecalendar app. Electronic calendar stays current whether you're at work or traveling—no manual updates needed.
  • All-In-One Home Assistant: Smart calendar builds habits and encourage independence with personalised chores. Display daily meal plan to reduce the daily ‘What's for dinner?’ stress. To-do lists keep track of everything you want. Turning chores into valuable achievements with stars and rewards, making life easier and more enjoyable.
  • Manage Anytime Anywhere: BSIMB electronic calendar for family stay connected with the powerful ecalendar app. Access and manage your digital wall calendar anytime, anywhere with free features like stars & rewards, photo screensaver, chore charts, meals plans, to-do lists etc. Electronic planner paid features like Magic Import tailor advanced tools for users with special needs.
  • Stay in Touch with Loved Ones: Share photos or videos from your phone to electronic wall calendar with app, instantly update your life moment, and keep in touch with your family, especially those who are far away; you can also surprise your smart calendar digital wall by sending him a wish card; it's also great for keeping tech-illiterate elders in touch with the youngsters.
  • Container-instance health: Poll or query DescribeContainerInstances with CONTAINER_INSTANCE_HEALTH. Record overallStatus, individual detail entries, and statusReason.
  • Health-change events: Watch ECS container-instance health change events so an impairment, transition to DRAINING, and replacement can be correlated on one timeline.
  • Task placement and stopping: Check service events and task stop reasons to distinguish a host drain from an application crash or deployment replacement.
  • GPU telemetry: Enable Container Insights enhanced observability for GPU-enabled Managed Instances when you need NVIDIA DCGM metrics at container, task, and instance levels. Basic Container Insights does not collect these GPU metrics.
  • Capacity availability: Check whether the provider can launch the requested GPU type in the selected subnets and Availability Zones. A correct repair setting cannot create capacity that a quota, subnet, or regional shortage prevents.

Keep the XID, health-status transition time, instance identifier, task identifiers, and replacement identifier together in the incident record. That information lets you determine whether the event was a recurring GPU/driver issue, a one-off host failure, or a capacity-placement problem during recovery.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Configuration workflow in the ECS console

The exact console wording can change, but the required decisions remain the same. Follow this order so the provider is usable before repair is enabled:

  1. Open Amazon ECS and select the cluster that will run the GPU workload.
  2. Start the flow to create a capacity provider and choose Managed Instances, not an Auto Scaling group-backed provider.
  3. Provide the infrastructure role and instance profile required for Managed Instances.
  4. Set the VPC, subnets, security groups, and other network parameters needed for ECS agent communication and image access.
  5. Define instance requirements that include the required GPU accelerator type and count, or select GPU instance types that satisfy the workload.
  6. Set the managed provider’s automatic-repair action to ENABLED unless your change-control policy requires monitoring without automatic replacement.
  7. Create the provider, attach it to the cluster, and verify its returned configuration.
  8. Configure the service’s capacity-provider strategy to use this provider. A service that never places tasks on the Managed Instances provider cannot benefit from its replacement behavior.
  9. After instances register, verify the ECS agent version and query container-instance health with DescribeContainerInstances.

If you are updating an existing provider, change only the provider setting required for the repair policy, read the provider back, and confirm that the intended GPU requirements and network settings remain intact.

Troubleshooting common failure modes

The provider reports health, but no replacement occurs

First check autoRepairConfiguration.actionsStatus. DISABLED intentionally preserves monitoring while suppressing the repair action. Next confirm that the instance is actually IMPAIRED, rather than INSUFFICIENT_DATA or INITIALIZING. Review ECS health-change events and the provider configuration returned by the API rather than relying on an old console view.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

The instance is draining very slowly

Task stop timeouts are honored. A workload with a long stop timeout, a process that does not handle termination, or a task that cannot shut down cleanly can delay completion. Check service events and task stop reasons, then adjust application shutdown behavior and timeout values deliberately; do not assume ECS has frozen.

A replacement is requested but does not launch

Inspect GPU quotas, subnet IP capacity, Availability Zone placement, instance-type availability, security-group egress, image access, and the Managed Instances roles. The 20-percent concurrency limit can also make recovery appear serialized, especially when the provider has fewer than nine instances.

The health response has no useful GPU detail

Ensure the query includes CONTAINER_INSTANCE_HEALTH and inspect the complete details array, not only overallStatus. Confirm the agent is version 1.57.0 or later. Use enhanced Container Insights for DCGM telemetry; basic Container Insights is not a replacement for GPU metrics.

Only one service task is replaced, while the host remains healthy

That pattern normally indicates a task-definition health-check failure or another task-level event, not a GPU instance impairment. Compare the task’s health status and stop reason with the container-instance health response before treating it as hardware failure.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

What changes when the cluster uses ordinary EC2 capacity

If your ECS cluster uses an EC2 Auto Scaling group or another non-Managed-Instances capacity provider, the Managed Instances GPU auto-repair procedure described here is not established for that setup. ECS agent connectivity checks, EC2 status checks, Auto Scaling replacement policies, custom CloudWatch alarms, and instance termination workflows are neighboring controls, but they should not be mislabeled as the ECS Managed Instances DCGM check.

Identify the actual compute provider first. Then use the health and recovery controls documented for that provider, and test how a failed GPU instance affects task draining, replacement capacity, and service scheduling. Do not copy the autoRepairConfiguration object into an unrelated provider and expect it to create the same behavior.

Production readiness checklist

  • Managed Instances is explicitly selected as the capacity provider.
  • GPU accelerator type/count or GPU instance requirements match the task’s CUDA workload.
  • Infrastructure role, instance profile, networking, quotas, and image access are ready.
  • ECS agent version is 1.57.0 or later.
  • actionsStatus is read back and confirmed as ENABLED when automatic replacement is required.
  • Services use a capacity-provider strategy that places tasks on the GPU provider.
  • Task stop timeouts and application shutdown handlers allow graceful draining.
  • Health-change events and DescribeContainerInstances output are collected for incident response.
  • Enhanced Container Insights is enabled when container-, task-, and instance-level DCGM metrics are needed.
  • Capacity planning accounts for the 20-percent concurrent-drain limit and the one-at-a-time limit on providers with fewer than nine instances.
  • Runbooks distinguish task health failures, ECS agent issues, EC2 status failures, and GPU XID events.

The Bottom Line

For automatic GPU host recovery, run the workload on an Amazon ECS Managed Instances capacity provider, verify GPU requirements and agent support, and set managedInstancesProvider.autoRepairConfiguration.actionsStatus to ENABLED. ECS then uses DCGM-backed health signals, drains impaired instances, launches replacements, honors task stop timeouts, and terminates the failed hosts within its documented concurrency limits.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Amazon ECSauto repairAWScontainer healthDCGMGPUManaged Instances

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Recommended PC Tool
Recommended PC Tool
PC Slower Than It Used to Be?Free scan - under a minute
Crashes, No Sound, or Screen Glitches?Free driver scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.