Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
A GPU-cloud outage can stop a running job, block access to replacement GPUs, and erase unsaved work—but it does not automatically destroy your data. The outcome depends on the failure scope and on where your code, datasets, checkpoints, logs, secrets, and model artifacts are stored. Data kept only on instance-local storage may disappear during host maintenance or a restart, while independently stored copies can survive a compute outage. Service-level agreements (SLAs) may provide credits or define response targets, but they do not guarantee recovered compute, restored data, or uninterrupted business operations. A defensible design separates compute recovery from data recovery, sets explicit recovery time and recovery point objectives, verifies that backups can be restored, rehearses failover, and confirms where substitute GPU capacity will come from.
What can fail when a GPU cloud provider has an incident?
“The provider is down” can describe several technically different events. Each has a different blast radius and recovery path, so identify the failure before choosing a response.
As an Amazon Associate I earn from qualifying purchases.
Recommended Free Tools
| Failure scenario | Likely effect on workloads | Questions to answer |
|---|---|---|
| GPU hardware fault | A VM may remain reachable but return CUDA, driver, or collective-communication errors; a training run can stall or terminate. | Can the GPU be reset, or must the VM be rebooted or replaced? Is the fault isolated to one host? |
| Host or zone incident | Running instances can stop and capacity in that location may be unavailable. | Are equivalent GPUs available in another zone or region, and can your networking and quotas follow? |
| Network outage | Workers, object storage, registries, control planes, or users may be unable to communicate even if GPUs are powered on. | Which dependency is unreachable, and is there an independent network path? |
| Storage outage | Jobs may fail to read datasets or write checkpoints; a local disk may be inaccessible or lost. | Do you have a separate, verified copy of the required data and the latest checkpoint? |
| Provider control-plane incident | Existing VMs might continue running while provisioning, authentication, monitoring, or management operations fail. | Can you restart, scale, rotate credentials, or obtain support without the control plane? |
| Account suspension or quota failure | Instances can be terminated or new capacity denied even when the platform itself is healthy. | Who can appeal, and do you have an independent account or provider? |
| Provider business failure | Access, support, billing, and data-export windows may all become uncertain. | How quickly can you retrieve data and recreate the environment elsewhere? |
A short inference interruption, a week-long distributed training run, and a production model-serving endpoint therefore need different safeguards. Record the exact incident scope and affected service before assuming that a restart, a region change, or a provider move will work.
Will you lose training data if the GPU instance stops?
Not necessarily. Data durability and compute availability are separate properties: a VM can stop while data in independent persistent storage remains available, but data written only to temporary local storage can be lost.
#1 Best Overall
- Get NVMe solid state performance with up to 1050MB/s read and 1000MB/s write speeds in a portable, high-capacity drive(1) (Based on internal testing; performance may be lower depending on host device & other factors. 1MB=1,000,000 bytes.)
- Up to 3-meter drop protection and IP65 water and dust resistance mean this tough drive can take a beating(3) (Previously rated for 2-meter drop protection and IP55 rating. Now qualified for the higher, stated specs.)
- Use the handy carabiner loop to secure it to your belt loop or backpack for extra peace of mind.
- Help keep private content private with the included password protection featuring 256‐bit AES hardware encryption.(3)
- Easily manage files and automatically free up space with the SanDisk Memory Zone app.(5). Non-Operating Temperature -20°C to 85°C
Why instance-local disks are risky
Google Cloud states that GPU instances stop during host maintenance. When a Local SSD is attached, its data is lost after that stop; Google says it cannot recover the data after a host-maintenance restart and advises, Avoid storing data with strong persistency requirements on Local SSD.
That guidance applies to Google’s Local SSD product, not to every provider’s storage, but the design lesson is general: treat instance-local disks as scratch space unless the provider explicitly guarantees persistence.
Keep source code, environment definitions, datasets, checkpoints, logs, model artifacts, job specifications, and secrets in storage whose lifecycle is independent of the GPU host. A checkpoint copied to a separate storage service before a host failure is materially different from a checkpoint that exists only on the VM’s boot or local scratch disk.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesWhat “recoverable” should mean
A copy is not a recovery plan until you can retrieve it with the required identity, network access, permissions, and tooling. Verify that files are complete and readable, record checksums or equivalent integrity information, and perform a restore into a clean environment. Test the most recent checkpoint format, not merely an old sample dataset.
What does a GPU-cloud SLA actually guarantee?
An SLA is a contract for a named service, measurement method, exclusions, claims process, and remedy. It is not a universal promise that your workloads or data will be restored.
Read the service and measurement scope
NVIDIA’s Cloud Services Service Level Agreement, last modified November 5, 2025, calculates availability over a calendar month and points to service-specific terms. Its general exclusions include force majeure and issues outside NVIDIA’s reasonable control. A customer claim must include incident details and reach NVIDIA within two months of the incident’s start.
Rank #2
- MADE FOR THE MAKERS: Create; Explore; Store; The T7 Portable SSD delivers fast speeds and durable features to back up any endeavor; Build your video editing empire, file your photographs or back up your blogs all in an instant
- SHARE IDEAS IN A FLASH: Don’t waste a second waiting and spend more time doing; The T7 is embedded with PCIe NVMe technology that brings fast read and write speeds up to 1,050/1,000 MB/s¹, making it almost twice as fast as the T5
- ALWAYS MAKE THE SAVE: Compact design with massive capacity; With capacities up to 4TB, save exactly what you need to your drive – from large working files to game data and everything in between
- ADAPTS TO EVERY NEED: Whether using a PC or mobile phone, count on the T7 for extensive compatibility²; It’s a true team player when it comes to heavy-duty application usage or file-saving
- HI RESOLUTION VIDEO RECORDING: Record Ultra High Resolution (4K 60fs) videos directly onto the T7 Portable SSD with your favorite camera or mobile devices; Supports iPhone 15 Pro Res 4K at 60fps video and more³
For a critical issue, the same SLA says NVIDIA will use commercially reasonable efforts to find a resolution within 24 hours after determining that the incident is critical. The document explicitly describes this as a target that may not always be achieved, not a guaranteed time to recover your workload. The SLA lists a 99% monthly service-availability target for NVIDIA Omniverse Cloud; that figure belongs to that particular offering and should not be generalized to GPU compute.
Credits are not restored operations
Any remedy depends on the applicable offering and its terms. A service credit can compensate for an availability shortfall, but it does not recreate lost checkpoints, provide equivalent GPUs during a shortage, pay for delayed business activity, or remove the engineering work of rebuilding an environment. Before relying on an SLA, record the covered component, uptime calculation, exclusions, severity definitions, response targets, claim deadline, evidence requirements, and exact remedy.
How should you set recovery time and recovery point objectives?
Set an RTO for each workload: the maximum acceptable time before service resumes. Set an RPO for each workload: the maximum acceptable amount of recent work or data that may be lost. These are business decisions, not automatic properties of a GPU service.
| Workload | RTO question | RPO question | Design implication |
|---|---|---|---|
| Interactive inference API | How long can requests be unavailable before users are affected? | Can the service use the last published model? | Keep a deployable model artifact and configuration outside the serving VM; maintain a tested alternate endpoint or region if the RTO is short. |
| Batch inference | When must the batch finish? | Can completed partitions be retained and retried? | Persist inputs, partition status, and outputs separately so only incomplete work is rerun. |
| Long training run | How quickly must training resume? | How many minutes or hours of training can be repeated? | Write versioned checkpoints at a cadence that fits the RPO, and validate that a new environment can load them. |
| Distributed research job | Can the full cluster be recreated elsewhere? | Which optimizer state, random seeds, and logs are required? | Save the complete launch configuration and rendezvous settings, not only the model weights. |
NVIDIA’s Requirements for AI Clouds, version 2.4 calls for defined and auditable RTO and RPO values, backups with restore-integrity verification, failover testing on a defined cadence with recorded results, and assigned disaster-recovery roles. Those are requirements in that provider context; they are not an automatic commitment for every GPU-cloud customer.
What should you do during an outage?
Use a written runbook so an incident does not depend on one engineer remembering undocumented commands or credentials.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Rank #3
- Capacity Display Variance: 500GB external ssd often appears as around 465GB on Windows. MacOS can show full 500 GB capacity. This is binary calculation difference and doesn’t affect SSD hard drive actual physical storage
- 1050 MB/s Speed: Instantly access to your files with blazing-fast 10Gbps external SSD read up to 1050MB/s and write up to 1000MB/s. LED Light indicates USB SSD instant activity
- Data Security: Solid state drives S.M.A.R.T. health diagnostics and adaptive TRIM optimizing data block management ensures consistent write speeds and extends the longevity of the portable SSD
- USB-C & USB-A Cable: Both cables featuring rapid USB 3.2 Gen2, this USB SSD effortlessly bridges devices, enabling seamless cross-platform file transfers and backup between computers, smartphones, tablets and iPhone
- Always Fast: No slowdowns for large file transfers. With SLC caching (25% of current available capacity allocated as high-speed cache), this external SSD delivers steady 10Gbps for transfers within the cache capacity
- Classify the incident. Identify the affected project, instances, GPU type, region or zone, start time, and whether the symptom is hardware, host, network, storage, control-plane, account, or business-related.
- Protect evidence. Save scheduler events, job IDs, timestamps, application logs, container digests, recent configuration changes, and the last known-good checkpoint before repeatedly restarting anything.
- Check durable copies. Confirm that the required dataset, checkpoint, model artifact, environment definition, and secrets are available outside the affected compute instance. Check object counts, checksums, and access permissions.
- Attempt the smallest safe recovery. For a single-GPU fault, follow the provider’s documented reset or reboot procedure. For a host or zone incident, request replacement capacity or move to an approved location only after confirming quotas and networking.
- Recreate the environment. Use a pinned container image or reproducible build, install the recorded driver and framework versions, restore credentials through the approved secret path, and validate storage and interconnect access.
- Resume from a verified checkpoint. Confirm that optimizer state, tokenizer or preprocessing versions, random-seed requirements, and distributed-worker settings match the checkpoint’s assumptions.
- Record the incident and claim window. Note the provider ticket, severity classification, service covered by the SLA, and any deadline for a credit claim. For NVIDIA’s general SLA terms, the stated deadline is two months from incident start.
How do you troubleshoot a suspected GPU hardware fault?
Start with the platform’s diagnostic guidance rather than repeatedly deleting and recreating instances. Google Cloud’s GPU VM troubleshooting documentation directs operators to inspect kernel logs for numeric NVIDIA Xid errors.
Collect the diagnostic record
- Instance identifier, project, zone, GPU model, and suspected host or node.
- Complete NCCL logs when the failure involves distributed communication.
- Health-check output, failing commands, kernel and diagnostic logs, and timestamps in UTC.
- Recent driver, container, framework, networking, or workload changes.
Depending on the GPU instance family and the fault, Google’s prescribed recovery can include resetting the GPU or rebooting the VM. A reset may preserve the operating system context but can terminate work using that device; a reboot is more disruptive and still cannot recover data that existed only on lost local storage. Follow the provider’s current procedure for the specific instance family.
Can you move an AI workload to another region or provider?
Sometimes, but portability is an engineering property you must test—not a guaranteed feature of a GPU cloud. A move succeeds only when the software environment, data, infrastructure configuration, required GPU capacity, and operational access can be recreated under incident conditions.
| Recovery path | Strength | Hard constraint to test |
|---|---|---|
| Same provider, another region | Existing identity, storage formats, and APIs may reduce rebuild time. | The needed GPU model, quota, network topology, and regional storage access may be unavailable. |
| Second provider | Reduces dependence on one provider’s control plane and business operations. | Drivers, container runtime, orchestration, images, IAM, networking, and GPU interconnect behavior may differ. |
| Local or hybrid fallback | Provides an independent execution location for critical jobs. | On-premises or reserved hardware may not match cloud scale, and keeping it current adds cost and operational work. |
Before an incident, test a clean launch with the same code, pinned dependencies, infrastructure-as-code, datasets, checkpoints, secrets process, and monitoring. Measure data-transfer time and cost, including any egress or access restrictions. A plan that says “use another provider” without verified credentials, quota, capacity, and restore steps is an intention, not failover.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →What should a durable backup and failover design contain?
Keep an inventory that distinguishes what is persistent, replicated, and actually recoverable.
Rank #4
- NEARLY 2X FASTER THAN OUR PREVIOUS GENERATION(8) – move 1,000 high-res photos in under 60 seconds(6) with up to 2000MB/s transfer speeds(2).
- IP65 RATING AND UP TO 3M DROP PROTECTION(3) – protects against spills and drops.
- POCKET-SIZED – fits easily in pockets and small bags.
- SPACE TO OWN YOUR AI CONTENT – speed and capacity to download your high-res clips and photo edits.
- 256-BIT AES ENCRYPTION(4) – helps keep private files secure with password protection.
- Code and environment: source repositories, lockfiles, container definitions, build instructions, driver and framework compatibility notes, and infrastructure configuration.
- Data: source datasets, preprocessing outputs, labels, manifests, and the permissions needed to read them.
- Training state: model weights, optimizer state, scheduler state, tokenizer files, random seeds, epoch or step counters, and the exact command used to launch training.
- Operations: logs, metrics, alerts, job metadata, network policies, quotas, DNS settings, and runbooks.
- Security: secret references and a tested method for issuing replacement credentials without copying secrets into images or logs.
Replicate important items outside the failure domain, verify integrity automatically, and perform a full restore at a defined cadence. An external SSD can be an additional offline copy for artifacts or data, but it is not automatic replication, alternate compute, or protection from a site-wide event when it remains in the same location.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How do component-level SLAs change the risk?
A provider may guarantee one component while excluding the GPU servers themselves. KDDI GPU Cloud’s published SLA and availability table illustrates this distinction: it lists 99.95% monthly availability for high-speed storage and the service network, while marking GPU servers outside that availability guarantee. The same page gives a separate hardware-replacement time commitment under stated conditions.
Do not multiply a storage or network percentage into an assumed GPU-compute guarantee. Map every dependency in your workload—GPU host, scheduler, image registry, object storage, high-speed fabric, identity service, DNS, and support channel—to the term that actually covers it.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallIs NVIDIA Recovery a production solution for a failed GPU cloud?
Not on the evidence available in its current documentation. NVIDIA’s NVIDIA Recovery page labels the project experimental and says it is not ready for production use. It manages workloads rather than repairing hardware or storing checkpoints, and Kubernetes-native workload recovery is not implemented there.
Use it only after independently checking its current status, supported environments, and boundaries. A production plan still needs durable checkpoint storage, replacement capacity, tested orchestration, and an operational owner.
Best Value
- Capacity Display Variance: 1TB external ssd often appears as around 931GB on Windows. MacOS can show full 1 TB capacity. This is binary calculation difference and doesn’t affect SSD hard drive actual physical storage
- 1050 MB/s Speed: Instantly access to your files with blazing-fast 10Gbps external SSD read up to 1050MB/s and write up to 1000MB/s. LED Light indicates USB SSD instant activity
- Data Security: Solid state drives S.M.A.R.T. health diagnostics and adaptive TRIM optimizing data block management ensures consistent write speeds and extends the longevity of the portable SSD
- USB-C & USB-A Cable: Both cables featuring rapid USB 3.2 Gen2, this USB SSD effortlessly bridges devices, enabling seamless cross-platform file transfers and backup between computers, smartphones, tablets and iPhone
- Always Fast: No slowdowns for large file transfers. With SLC caching (25% of current available capacity allocated as high-speed cache), this external SSD delivers steady 10Gbps for transfers within the cache capacity
What should you verify before you depend on a GPU provider?
Ask for concrete answers rather than a headline uptime percentage:
- Which exact GPU, storage, network, control-plane, and support services are covered?
- How is availability measured, and what causes are excluded?
- What severity levels, response targets, restoration targets, and escalation contacts apply?
- What evidence and claim deadline apply, and is the remedy a credit or something more specific?
- Can you export datasets, checkpoints, images, logs, and configuration in a usable format?
- What substitute GPU models and regions have realistic capacity during a broad incident?
- How are maintenance stops handled, and which attached storage survives them?
- When was the documentation last modified, and how will changes be communicated?
Provider capacity, SLAs, recovery features, and technical documentation change. Recheck the applicable terms and platform guides when you design or materially change a workload.
Frequently Asked Questions
Does a higher uptime percentage mean my model data is safer?
No. Availability measures whether a defined service is reachable during the measurement period; it does not establish backup durability, checkpoint frequency, or successful restoration. Evaluate storage retention and restore testing separately from compute uptime.
How often should I run a disaster-recovery exercise for GPU jobs?
Set a cadence that matches the rate of change and the workload’s RTO/RPO, then rerun the exercise after major changes to images, drivers, orchestration, identity, or storage. Record the result, elapsed recovery time, missing prerequisites, and corrective owner rather than treating a one-time drill as proof.
Are model-weight checkpoints sufficient to restart distributed training?
Often they are not. A restart may also require optimizer and scheduler state, tokenizer and preprocessing versions, worker topology, random-seed details, and the exact software environment; omit those and the job may resume with different behavior or fail to load.
What is the first thing to ask a provider’s support team during a GPU incident?
Ask which service component and failure domain are affected, whether the incident is classified as a hardware fault or capacity outage, and whether replacement GPUs are available in an approved region. Give the ticket the instance identifiers, timestamps, diagnostic logs, failing commands, and recent changes so the provider can correlate the failure quickly.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
