Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesNo. Amazon ECS GPU auto-repair for Managed Instances replaces an impaired GPU host; it does not live-migrate a running task or copy the host’s memory and local files to the replacement. ECS drains the instance, gives containers their configured shutdown window, starts replacement service tasks when scheduling rules and capacity permit, and then terminates the old instance. Data stored only on that host is not preserved by the repair process.
Scope: the ECS feature covered here
This behavior applies to GPU auto-repair for Amazon ECS Managed Instances. It should not be generalized to every ECS cluster running on ordinary customer-managed EC2 instances. Managed Instances use NVIDIA Data Center GPU Manager (DCGM) to monitor GPU health. Critical failures, including classes of Xid errors documented by AWS, can cause ECS to mark the instance impaired and begin replacement.
As an Amazon Associate I earn from qualifying purchases.
Auto-repair is an infrastructure recovery mechanism, not a checkpointing or migration system. A container may receive a graceful stop signal, but its process, GPU context, in-memory state, and files that exist only on the old host do not move to the new host.
Free tools Windows power users keep installed
One-click scans. No signup required.
What happens when a GPU instance is impaired
The sequence is a drain-and-replace operation:
- DCGM reports a critical GPU condition. ECS evaluates GPU health using NVIDIA DCGM. The AWS GPU auto-repair documentation lists the complete set of critical conditions; examples include a GPU that has stopped processing, uncorrectable ECC errors, a GPU falling off the bus, and NVLink errors.
- ECS marks the Managed Instance impaired. The scheduler treats the host as unsuitable for continued operation and changes its state to
DRAINING. - Tasks are given a stop opportunity. ECS follows normal task-shutdown behavior. For Linux containers, it sends the configured stop signal, which is SIGTERM by default, then sends SIGKILL after the configured
StopTimeoutif the process has not exited. The timeout limits how long the host can wait; it does not turn the task into a migrated task. - Service scheduling makes replacement decisions. For tasks owned by an ECS service, the scheduler stops old tasks and places replacement tasks according to the service’s deployment settings and available capacity.
- ECS provisions a replacement Managed Instance. The replacement is a different host. It has its own local filesystem, instance-store devices, GPU state, and attached resources.
- The impaired instance is terminated. Anything that exists only on that host is subject to the normal consequences of instance termination or hardware failure.
Because the old task is stopped and a new task is scheduled elsewhere, an application can experience interruption even when the replacement starts quickly. ECS does not document transparent live migration of GPU contexts or container memory.
#1 Best Overall
- [ Maximum AI Compute Power ] Dominate complex workloads with the ASUS ESC8000A-E13. This 4U rack server is a powerhouse engineered for mass-scale AI, machine learning, and deep training. Featuring support for dual AMD EPYC 9005/9004 processors and up to eight dual-slot GPUs, it delivers the raw computational muscle required to train LLMs and run complex simulations effortlessly. Accelerate your data science pipeline and transform raw data into actionable intelligence faster than ever.
- [ Advanced Thermal Efficiency ] High performance demands elite cooling. The ESC8000A-E13 features a cutting-edge aerodynamic design with independent CPU and GPU airflow tunnels. Equipped with redundant hot-swap fans and optimized for liquid cooling integrations, this 4U server ensures maximum uptime under heavy, sustained workloads. Keep your data center running cool, quiet, and highly efficient while preventing thermal throttling during mission-critical enterprise operations.
- [ Scale with Flexible Storage ] Future-proof your infrastructure with unmatched storage and expansion flexibility. This offers comprehensive front-panel drive bays supporting Gen5 NVMe, SAS, or SATA drives alongside multiple PCIe 5.0 slots. Designed as a high-density 4U server capable of housing eight dual-slot GPUs: NVD H200, RTX PRO 6000 Blackwell, RTX PRO 4500 Blackwell or AMD Instinct MI350P PCIe Card, each supporting up to 600 watts.
- [ Enterprise-Grade Reliability ] Minimize downtime and secure your ecosystem with server-grade redundancy. The ESC8000A-E13 is built for 24/7 continuous operation, boasting 2+2 redundant (3200W total) 80 PLUS Titanium power supplies and integrated ASUS ASMB11-iKVM for comprehensive out-of-band management. Ideal for cloud service providers, rendering farms, and large enterprise infrastructure, it combines robust physical hardware with smart remote monitoring to safeguard your digital assets.
- [Reliability Guaranteed] Shop with total peace of mind knowing that every new computer component we sell is backed by our EPC 3-year warranty. Whether you are investing in high-speed DDR5 RAM or a powerhouse GPU, we protect your build against defects and performance failures. We stand firmly behind the quality of our hardware, ensuring that your setup remains fast, stable, and secure for years to come.
How service tasks are replaced
minimumHealthyPercent controls the lower health bound
When a draining instance hosts service tasks, ECS tries to keep the service within its deployment constraints. minimumHealthyPercent determines how far the number of healthy running tasks may fall during replacement. A higher value protects serving capacity but may prevent ECS from stopping enough old tasks to free resources if the cluster is already tight.
maximumPercent determines whether overlap is possible
maximumPercent sets the upper limit for running and pending tasks during a deployment or replacement. If capacity is available and the value permits it, ECS can start replacement tasks before stopping every old task. That overlap can reduce interruption, but it is not a promise of zero downtime: a GPU replacement may take time to provision, capacity may be unavailable, and an impaired host may stop processing before a replacement is ready.
Capacity decides whether the desired sequence can occur
With restrictive percentages or insufficient GPU capacity, ECS may need to stop tasks on the draining host before it can place replacements. Plan for the possibility that a service temporarily runs below its desired count. Use capacity-provider scaling and service deployment settings together; changing percentages alone cannot create a missing GPU.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What happens to standalone tasks
A standalone task is not managed by an ECS service deployment controller. A manual instance drain does not automatically replace a standalone task; it can continue until it finishes or an operator stops it. GPU auto-repair is different because the impaired host is ultimately terminated. A standalone task on that host therefore loses its execution environment when termination occurs and is not automatically relaunched as a replacement task.
Rank #2
- NVIDIA Volta GV100 Architecture — 4,608 CUDA Cores, 640 1st-Gen Tensor Cores delivering 14 TFLOPS FP32 and 112 TFLOPS deep learning performance for AI training, inference, HPC, and scientific computing workloads
- 32GB HBM2 ECC Memory — 900 GB/s Bandwidth — High-bandwidth memory on a 4096-bit bus with ECC error correction provides the memory capacity and throughput required for the largest AI models, simulations, and datasets
- PCIe 3.0 x16 Interface — 250W TDP — Standard PCIe Gen3 connectivity with passive cooling designed for enterprise rack server deployment in HPE ProLiant, Dell PowerEdge, and Supermicro platforms with adequate chassis airflow
- NVLink — Scale to 96GB Unified Memory — Connect two V100 GPUs via NVLink at 300 GB/s bi-directional bandwidth to scale GPU memory from 32GB to 96GB for larger AI training and HPC workloads
- Multi-Precision Computing — Supports FP64 (7 TFLOPS), FP32 (14 TFLOPS), FP16 (112 TFLOPS) and INT8 precision modes for flexible deployment across training, inference, and scientific simulation workloads
If a batch or inference job must resume, build that behavior outside the host: submit it through a service, queue, workflow, or external controller, and persist checkpoints somewhere independent of the instance. Treat a standalone task on a Managed Instance as interruptible, even if it is currently in RUNNING state.
Does local data survive replacement?
ECS does not synchronize storage between EC2 instances. A file being present on the impaired host says nothing about whether a replacement task can see it. The relevant question is not merely whether data survives a task restart, but whether it survives host termination and can be attached or reached by a task on another host.
| Storage or data location | After task stop | After host replacement | Available to a replacement task? | Important lifecycle detail |
|---|---|---|---|---|
| Container writable layer | Normally no durable guarantee | No | No, unless copied elsewhere | It belongs to the task’s container filesystem. |
Bind mount using an EC2 sourcePath |
Files remain on that host while the host exists | No portability guarantee | Not on a different instance | The path can persist on the original EC2 host, but ECS does not replicate it before termination. |
| Instance-store volume | May remain while the same instance remains running | No | No | AWS states that instance-store data is lost when the instance is stopped, terminated, or experiences hardware failure. |
| Amazon EBS volume | Depends on attachment and task mode | Depends on how it is reattached and its deletion settings | Potentially, after successful attachment | Persistence is not automatic for every ECS EBS configuration; verify attachment, placement, and delete-on-termination behavior. |
| Amazon EFS | Yes, as a network file system | Designed to remain independent of one host | Yes, when the replacement task can mount it | Validate mount configuration, networking, permissions, and application consistency. |
| External durable database, object store, or queue | Independent of the task filesystem | Independent of the replaced host | Yes, subject to credentials and network access | Application-level durability, consistency, and retry behavior still apply. |
Bind mounts: host persistence is not replacement durability
In an EC2 launch type task definition, an explicit bind-mount sourcePath points to a directory on the host. According to ECS bind-mount configuration guidance, that directory can retain files on that particular EC2 instance. However, the replacement Managed Instance has a different filesystem and does not receive a synchronized copy. ECS classifies bind mounts as ephemeral in its storage comparison; the narrower exception is that an explicit source path can persist for the life of the original host.
Instance store: assume loss
Instance-store devices are physically tied to the instance. AWS states in its Managed Instances architecture documentation: “Instance store volumes are ephemeral. Data on instance store volumes is lost when the instance is stopped, terminated, or when hardware fails. Do not use instance store for data that must persist.” GPU auto-repair ends with termination of the impaired instance, so instance-store-only checkpoints, caches, and intermediate files should be treated as lost.
Rank #3
- AI-Optimized: Designed to support up to 4 GPUs, it is perfect for handling intensive AI and machine learning tasks, ensuring high performance and scalability for advanced computational needs.
- Intelligent Storage: Equipped with 8 hot-swappable 3.5" SATA/SAS drives (12Gbps), featuring SGPIO and temperature control, it ensures efficient data management and reliable storage performance.
- Robust Cooling: The system includes 3x 12038 hot-swap PWM fans and 2x 8038 rear fans, providing advanced thermal management to maintain optimal temperatures and ensure stable operation under heavy workloads.
- Rack-Ready: Comes with a pre-installed rail kit, allowing for quick and easy installation in standard 19-inch server racks, making it ideal for data center environments and enterprise setups.
- Versatile Connectivity: Offers USB 3.0 and the latest USB 3.2 Type-C ports, ensuring high-speed data transfer and compatibility with a wide range of peripherals and devices for enhanced connectivity options.
EBS and EFS require an explicit lifecycle design
ECS storage documentation identifies EBS and EFS as options, but they behave differently. EBS is a block volume with attachment and deletion rules. For ECS-managed EBS volumes, consult the current EBS volume configuration documentation and verify whether the volume is recreated, retained, or deleted for the task and deployment mode you use. Do not assume that every EBS volume automatically follows a replacement task.
EFS is network-attached and intended to remain available independently of one EC2 host, but the replacement task still needs a valid mount target, security-group access, IAM authorization where applicable, and an application that can tolerate concurrent access and partial writes.
Graceful shutdown: useful, but not continuity
A stop window gives an application time to finish a small unit of work, flush buffers, write a checkpoint, or release resources. It does not preserve an active GPU kernel or keep the same process alive on the replacement instance. Configure the container’s stop signal and StopTimeout deliberately, then make the process handle SIGTERM promptly. If it ignores the signal or cannot complete within the timeout, ECS sends SIGKILL and the remaining in-memory state is lost. No shutdown setting can guarantee that every workload will checkpoint successfully.
Auto-repair settings and monitoring
AWS enables GPU auto-repair by default for Amazon ECS Managed Instances. The setting is represented by actionsStatus inside autoRepairConfiguration, documented in the ECS API reference. Setting actionsStatus to DISABLED stops automatic replacement actions while ECS continues monitoring GPU health.
Rank #4
- Professional AI & Creator Workstation: AMD Radeon AI PRO R9700 GPU with 32GB GDDR6 is engineered for AI development, professional content creation, and compute-intensive workloads.
- Massive 32GB Memory Capacity: 32GB of GDDR6 memory on a 256-bit bus provides ample bandwidth for large AI models, 8K video editing, and complex 3D rendering.
- Advanced RDNA 4 with AI Accelerators: 64 Compute Units with 3rd Gen Ray Tracing and dedicated 2nd Gen AI Accelerators for groundbreaking AI performance and visual computing.
- Professional Blower Cooling: Efficient single blower design exhausts heat directly out of the chassis, ideal for multi-GPU workstation and server configurations.
- Enterprise-Grade Thermal Solution: Vapor chamber heatsink with industrial Honeywell PTM7950 thermal interface material ensures reliable cooling under sustained professional loads.
Disabling GPU auto-repair has a related trade-off: AWS documents that it also disables managed-daemon auto-repair. You may choose this during an investigation or when an external remediation system owns replacement, but you then assume responsibility for detecting the impairment, draining safely, and restoring capacity.
Use the linked GPU auto-repair page for the complete DCGM critical-code list rather than building automation around a partial list of Xid examples.
How much repair concurrency can you expect?
AWS limits simultaneous draining within a capacity provider: “No more than 20% of the instances belonging to the capacity provider can be drained at a time. If there are fewer than 9 instances in the capacity provider, at most one instance is drained at a time.” This limit reduces the blast radius of concurrent repairs, but it is a rate limit, not an availability guarantee. A single impaired GPU host can still interrupt every task placed on it, and a small capacity provider may have only one host eligible for work while replacement capacity is being created.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchDesigning a GPU workload to survive replacement
- Keep execution state outside the host. Store durable inputs, outputs, checkpoints, and job status in EFS, an appropriately configured EBS design, or an independent database, object store, or queue. Do not make an instance-store directory or bind mount the system of record.
- Make work restartable. Divide long jobs into idempotent units, record completed units, and resume from the latest committed checkpoint after a task starts on a new host.
- Handle termination signals. On SIGTERM, stop accepting new work, finish or checkpoint the current unit when feasible, flush writes, and exit before
StopTimeoutexpires. - Use a service when automatic replacement is required. Configure the desired count,
minimumHealthyPercent, andmaximumPercentfor the interruption you can tolerate. A standalone task has no service controller to relaunch it. - Reserve realistic GPU capacity. Ensure the capacity provider can launch replacement GPU instances. Deployment percentages cannot overcome a shortage of compatible GPU capacity.
- Validate storage attachment on a new host. Confirm that the task definition, networking, permissions, volume lifecycle, and deletion settings all work when a task lands on a different instance.
- Observe both task and infrastructure events. Alert on impaired instances, draining state, stopped tasks, failed placements, and checkpoint lag so an interrupted job is visible rather than mistaken for a successful completion.
Practical answers
Will my ECS task restart if the GPU instance is replaced?
A service task is normally stopped and replaced according to deployment settings and capacity. A standalone task is not automatically relaunched by ECS merely because its host was repaired. In neither case does ECS resume the original process with its memory and GPU state.
Do ECS bind-mount files survive instance replacement?
Not as portable data. An explicit sourcePath can remain on the original host, but that host is the one being terminated and ECS does not copy the directory to its replacement.
Can ECS instance-store data survive GPU auto-repair?
No. Instance-store data is lost when the instance is terminated or hardware fails, which are both relevant outcomes of this repair flow.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
