What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
This is one author’s out-of-the-box benchmark, not a general ranking. Sho Tanaka’s write-up (DEV Community, September 2026) used a single 3B-parameter model, one four-GPU node, and one run per configuration, with DeepSpeed left at its defaults. Within that setup, FSDP2 with full parameter resharding ran about 1.83x faster than DeepSpeed ZeRO-3 on four A100 40GB GPUs linked by NVLink. On four L4 24GB GPUs connected over PCIe, the order changed: ZeRO-3 finished fastest, and two of the ZeRO runs completed only after an allocator setting was enabled.
What follows covers the test conditions, the throughput and memory data, how to read the failures, a decision order for choosing a sharding setup, and how to get GPU nodes on Google Kubernetes Engine (GKE).
As an Amazon Associate I earn from qualifying purchases.
What the test covered
- Model and numerics: Qwen/Qwen2.5-3B in bf16, micro-batch size 1, sequence length 2,048.
- Hardware: four GPUs on a single GKE node. Two configurations were tested: four A100 40GB with NVLink, and four L4 24GB over PCIe.
- Frameworks: DeepSpeed ZeRO stages 0 to 3, ZeRO-3 with CPU offload, and three PyTorch FSDP2 modes: reshard, no-reshard, and reshard with CPU offload.
- Metric: median step time across 15 steps, reported as tokens per second.
- Tuning: defaults on both sides. The author states: “this is an out-of-the-box comparison — one model (3B), single node, n=1 (step times aggregated by median). DeepSpeed has tuning headroom (bucket sizes etc.) I did not explore; read this as a defaults-vs-defaults match.”
The FSDP2 modes map roughly onto the ZeRO stages. The author treats no-reshard as approximately ZeRO-2 and reshard as approximately ZeRO-3. In PyTorch’s FSDP2 API, the reshard choice corresponds to reshard_after_forward: with resharding, parameters are freed after the forward pass and gathered again for the backward pass, which saves memory at the cost of extra communication. Without it, they stay gathered. PyTorch’s FSDP2 getting-started guide (updated September 2, 2025) puts the core idea this way: “Comparing with DDP, FSDP reduces GPU memory footprint by sharding model parameters, gradients, and optimizer states.” Approximate equivalence is not identity, and the two modes differ in both memory and communication.
Throughput on A100 NVLink and L4 PCIe
| Configuration | 4 x A100 40GB, NVLink (tokens/s) | 4 x L4 24GB, PCIe (tokens/s) |
|---|---|---|
| FSDP2 no-reshard (approx. ZeRO-2) | 17,449 | 1,761 |
| FSDP2 reshard (approx. ZeRO-3) | 14,936 | 1,218 |
| FSDP2 reshard + CPU offload | 1,687 | Not collected in the write-up |
| ZeRO-1 | 16,642 | OOM |
| ZeRO-2 | 17,124 | 1,923* |
| ZeRO-3 | 8,151 | 2,290* |
| ZeRO-3 + offload | 2,615 | 1,405 |
| ZeRO-0 (no sharding) | OOM | Excluded (OOM noted) |
* Completed only in the successful reruns, with PYTORCH_ALLOC_CONF=expandable_segments:True set. See the allocator note below.
#1 Best Overall
- AI Performance: 767 AI TOPS
- OC mode: 2632 MHz (OC mode)/ 2602 MHz (Default mode)
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Axial-tech fan design features a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- A 2.5-slot design maximizes compatibility and cooling efficiency for superior performance in small chassis
Reading the NVLink result
On A100, ZeRO-1 (16,642 tokens/s), ZeRO-2 (17,124), and FSDP2 no-reshard (17,449) sit within about 5 percent of one another. The gap opens only when parameters are fully sharded: FSDP2 reshard reached 14,936 tokens/s against 8,151 for ZeRO-3, the ratio behind the headline figure. The write-up does not isolate a cause for that gap. Differences in communication scheduling and implementation details are the plausible candidates, but this single test cannot establish which one matters.
Reading the PCIe result
On the L4 machines the order reverses for full sharding. ZeRO-3 led at 2,290 tokens/s, ahead of ZeRO-2 at 1,923, FSDP2 no-reshard at 1,761, ZeRO-3 with offload at 1,405, and FSDP2 reshard at 1,218. The author suggests the tighter 24GB memory ceiling may favor deeper sharding. That is a hypothesis, and the write-up says so directly:
“This benchmark does not separate memory pressure from collective-scheduling and buffering effects, though, so the cause is not settled.”
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
The profiler data point the same way without settling it. The NCCL share of profiler time rose sharply on L4, but the author cautions that profiled kernel duration overlaps with compute, so it does not equal communication wait time.
| Configuration | NCCL share of profiler time, A100 | NCCL share of profiler time, L4 |
|---|---|---|
| FSDP2 no-reshard | 26.1% | 46.7% |
| ZeRO-2 | 17.3% | 50.2% |
Offload: memory saved, throughput paid
CPU offload moves optimizer state and related tensors off the GPU. In the A100 runs it traded memory for a steep speed penalty:
Rank #2
- Chipset: GeForce RTX 3050
- Boost Clock / Memory: 1492 MHz / 14 Gbps
- Video Memory: 6GB GDDR6
- Memory Interface: 96-bit
- Output: DisplayPort x 1 (v1.4a) / HDMI 2.1a x 2
| A100 configuration | Peak allocation, no offload | Peak allocation, CPU offload | Tokens/s, no offload | Tokens/s, CPU offload |
|---|---|---|---|---|
| FSDP2 reshard | 13.34 GB | 7.68 GB | 14,936 | 1,687 |
| ZeRO-3 | 18.18 GB | 6.69 GB | 8,151 | 2,615 |
Peak allocation figures are the author’s reported values for this workload and model. Offload reduced peak allocation by 42 percent for FSDP2 and 63 percent for ZeRO-3, while throughput fell about 8.9x for FSDP2 and about 3.1x for ZeRO-3. Those ratios come from one 3B model on A100 and should not be read as typical offload costs.
Decision order for choosing a sharding setup
The data support an order of operations more than a single winner. For your own model:
- Start with the lightest sharding that fits. In this test, ZeRO-1, ZeRO-2, and FSDP2 no-reshard delivered nearly identical throughput on A100, and full sharding cost throughput on NVLink. If your model fits under the lighter setting, this test gives no evidence that full sharding helps.
- If it does not fit, move to full sharding and choose by interconnect. On NVLink, FSDP2 reshard was the faster full-sharding option here. On PCIe L4, ZeRO-3 was faster, but its successful runs needed the allocator setting, so test both on your hardware before committing.
- Use CPU offload last. In the A100 runs it reduced peak allocation by 42 to 63 percent and cut throughput roughly threefold to ninefold.
- Validate on your workload before a long run. Measure with the model, precision, micro-batch size, and sequence length you will train with, and track the items below.
- Throughput and step time on your model, precision, batch size, sequence length, and GPU count.
- Peak allocated memory, and whether the job completes at all. A run that OOMs or times out produces no usable throughput.
- Interconnect and topology: NVLink versus PCIe, and single node versus multi-node. The write-up tested only single-node runs.
- NCCL profiler data, read with the caveat that communication overlaps compute.
- CPU offload memory savings set against its throughput cost.
- Software versions, allocator settings, launcher behavior, warmup and repetition count, and how much tuning each framework received.
When a run fails: OOM, node loss, or a watchdog timeout
The PCIe runs are the cautionary part of the write-up. The failure types look different in logs and call for different responses.
CUDA out-of-memory
This appears as a torch.OutOfMemoryError on a rank. One L4 ZeRO-3 rerun reported the error with 5.77 GiB reserved but unallocated. The author presents fragmentation as a likely factor, not a definitive diagnosis.
The error message suggested PYTORCH_ALLOC_CONF=expandable_segments:True, and with that setting the L4 ZeRO-2 and ZeRO-3 reruns completed. The author calls the option experimental and says it is not a universal fix for OOM or fragmentation. Use it as a diagnostic test. If a run still runs out of memory with it enabled, reduce memory use with deeper sharding or offload rather than relying on the setting.
Rank #3
- Powered by the NVIDIA Blackwell architecture and DLSS 4. System Requirements: Minimum 850W PSU with 16-pin 12V-2x6 (12VHPWR) connector required. Verify before purchasing.
- Military-grade components deliver rock-solid power and longer lifespan for ultimate durability. Compatibility: 348mm (13.7") length, 3.6 slots, 4.3 lbs. Confirm case clearance and slot spacing. GPU bracket included.
- Protective PCB coating helps protect against short circuits caused by moisture, dust, or debris
- 3.6-slot design with massive fin array optimized for airflow from three Axial-tech fans
- Phase-change GPU thermal pad helps ensure optimal thermal performance and longevity, outlasting traditional thermal paste for graphics cards under heavy loads
Node preemption or a lost node
Here a rank disappears mid-run. The write-up treats this as a log-investigation problem: identify which node went away, then check that node’s logs and the cluster events for the same window.
Collective watchdog timeout
The NCCL watchdog fired on an all-reduce of one element after 600,059 ms, roughly ten minutes. The author argues that one rank went silent before the collective, which is more consistent with a failure elsewhere than with a slow one-element operation. The practical rule follows: read the first error on any rank, not the watchdog message that comes after it. A timeout is often the symptom of an earlier failure on a different rank, and blaming interconnect bandwidth first sends the investigation in the wrong direction.
Launcher argument errors
The author also reports DeepSpeed launcher failures when the training script rejected an injected --local_rank=0 argument. The write-up’s workarounds were to make the script accept that argument or to pass --no_local_rank. Launcher behavior depends on version, so confirm it against the DeepSpeed release you have installed, using the official DeepSpeed “Getting Started” documentation (accessed October 7, 2026).
Getting GPU nodes on GKE
Google’s “Run GPUs in GKE Standard node pools” documentation (accessed October 7, 2026) states that GPU quota is required before you create GPU nodes and that GPU availability is specific to regions and zones. It maps A100 GPUs to the A2 machine series and L4 GPUs to G2. Quota and stock are separate questions, and the write-up shows why the distinction matters.
Quota is not stock
Quota is your project’s permission to use a given number of GPUs in a region. Stock is whether Google currently has physical capacity of that GPU type in a zone you can use. You can hold quota and still fail to create a pool. The write-up reports that repeated L4 pool creation failed in several zones despite quota, while an A100 Spot pool became available in a different GPU machine series. Its own four-GPU L4 stockout took about 14 hours to resolve. That describes one region and one time window, not a provisioning guarantee.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteSet up pools that can wait for capacity
- Confirm quota for the GPU type at least equal to the GPUs you plan to run. When autoscaling, count the maximum node count.
- Check which zones in your region list the accelerator, and choose the machine series: A2 for A100, G2 for L4.
- Use a regional cluster so the control plane is available across zones.
- Create a separate GPU node pool for each accelerator type. GPU nodes cannot be added to an existing node pool, so plan the pool before the job runs.
- Create each candidate pool with
--num-nodes 0, enable autoscaling, and set the minimum to zero. Pending GPU workloads can then trigger scale-up attempts. - Keep candidate pools in several zones so a stockout in one zone does not block the job.
- Confirm the GPU driver approach and GKE version constraints before creating the pool. The documentation describes automatic GPU driver installation where it is suitable.
Plan for interruptions
According to the same documentation, GPU nodes cannot be live migrated during maintenance events. Checkpoint at an interval that matches how much recomputation you can afford to lose, so that a maintenance event or a lost node costs you only the steps since the last checkpoint.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
