Verify cross-rail NCCL performance in layers: confirm Kubernetes has placed the job on the intended GPU nodes and exposed the expected network resources; measure local GPU paths; test the inter-node fabric; then run a multi-node NCCL collective and compare its per-rank results with hardware- and workload-specific expectations. A healthy single-node test does not establish that multiple nodes or rails are working.
What each test can establish
| Test layer | What it measures or checks | What it does not establish on its own |
|---|---|---|
| Kubernetes job and node checks | Whether the intended nodes, GPUs, operators, and network interfaces are available to the workload. | That GPU paths, fabric bandwidth, or NCCL collectives perform correctly. |
nvidia-smi topology and nvbandwidth |
Local GPU connectivity and GPU-to-GPU bandwidth. | Inter-node connectivity or cross-rail performance. |
ib_write_bw and ib_write_lat |
InfiniBand bandwidth and latency on the network devices selected for the test. | That an NCCL application uses every intended rail or achieves the same result under its workload. |
| Multi-node NCCL test | End-to-end collective correctness and performance across the participating nodes and GPUs. | Which layer is responsible for a slowdown without the lower-level checks and topology context. |
1. Confirm Kubernetes readiness and placement
Before interpreting performance, confirm that the job is actually running on the intended nodes and has access to the GPUs and network resources you mean to test. Check that both nodes are schedulable, the requested resources were assigned, and the workload is using the expected network interfaces rather than merely having a pod on each node.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
NVD RTX PRO 6000 Blackwell Professional Workstation Edition Graphics Card for AI, Design,... | $19,999.99 | Buy on Amazon |
| 2 |
|
NVIDIA RTX PRO 4000 Blackwell Graphics Card - 24GB GDDR7 ECC Memory, PCIe 5.0 x16, 4X DisplayPort... | $3,134.14 | Buy on Amazon |
| 3 |
|
PNY NVIDIA RTX A6000 | $5,945.17 | Buy on Amazon |
As an Amazon Associate I earn from qualifying purchases.
NVIDIA’s DGX Kubernetes validation example uses the MPI Operator, GPU Operator, and Network Operator for its multi-node NCCL workflow. It checks deployments with kubectl get deployment and verifies that the compute-side InfiniBand interfaces are up. Treat those as example checks, not universal installation instructions: operator names, namespaces, versions, and supported job templates depend on the cluster and its networking stack.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Record the node names, GPU allocation, network resource allocation, and relevant GPU, driver, CUDA, NCCL, and operator versions with each run. This makes later comparisons meaningful when placement or software changes.
#1 Best Overall
- ATTENTION BUYERS: EXPORTING this item outside the U.S. requires strict compliance with the Bureau of Industry and Security (BIS) U.S. Export Administration Regulations (EAR). Non-compliance carries fines up to $1 MILLION PER VIOLATION AND UP TO 20 YEARS IMPRISONMENT for both the buyer and seller. To uphold our regulatory obligations, we strictly follow BIS "Know Your Customer" guidance by screening each order and immediately aborting transactions showing unresolved regulatory red flags. ***
- [Double-Flow-Through Design] The RTX PRO 6000 Blackwell features a double-flow-through cooling design, optimizing efficiency and airflow to sustain peak performance under 600W power loads. | [5th Gen Tensor Cores] Deliver up to 3X the performance of the previous generation and support for FP4 precision for faster AI model processing times with reduced memory usage, enabling local fine-tuning of LLMs and generative AI | [4th Gen Ray Tracing Cores] Double the ray-triangle intersection rate of the previous generation to create photoreal, physically accurate scenes and immersive 3D designs with RTX Mega Geometry, which enables up to 100X more ray-traced triangles.
- [PCIe Gen 5] Support for PCIe Gen 5 provides double the bandwidth of PCIe Gen 4, improving data-transfer speeds from CPU memory and unlocking faster performance for data-intensive tasks like AI, data science, and 3D modeling. | [GDDR7 Memory] With 96 GB of GPU memory and 1.8 TB ps bandwidth, it can tackle massive 3D and AI projects, fine-tune AI models locally, explore large-scale VR environments, and drive larger multi-app workflows.
- [DisplayPort 2.1] Achieve unparalleled visual clarity and performance, driving high resolution displays at up to 8K at 240 Hz and 16K at 60 Hz. Increased bandwidth enables seamless multi-monitor setups while HDR and higher color depth support ensures superior color accuracy for precision work, such as video editing, 3D design, and live broadcasting.
- [NVIDIA Blackwell Streaming Multiprocessor] The new SM features increased processing throughput, and new neural shaders that integrate neural networks inside of programmable shaders | DLSS 4: Multi Frame Generation ensures ultra-smooth frame pacing for lifelike simulations. | [Universal MIG] Divide a single RTX PRO 6000 Blackwell into multiple isolated instances, each with dedicated resources, allowing for concurrent execution of multiple workloads, optimized GPU utilization, and secure isolation of different applications or users. [WARRANTY] 3 YR Manufacturer's Warranty. Bulk OEM Packaging. Retail Packaging is NOT included.
2. Establish local GPU and GPU-to-NIC paths
Inspect topology and peer access
On each participating node, run nvidia-smi topo -m to inspect the GPU and NIC topology. Use nvidia-smi topo -p2p n to inspect NVLink peer access, or nvidia-smi topo -p for PCIe peer access. These matrices show connectivity status; they do not measure the bandwidth a workload achieves.
Measure GPU-to-GPU bandwidth
Use NVIDIA’s nvbandwidth to measure the local GPU-to-GPU paths relevant to the node’s topology. Keep the result with the GPU model, topology, and software versions. A slow or failing local path should be resolved before attributing a multi-node slowdown to the fabric.
Check direct GPU-to-NIC communication
If the intended path uses GPU Direct RDMA (GDRDMA), verify that the NIC and driver support the configured path. NVIDIA documents nvidia-peermem as one way to enable peer-memory access; supported DMA-BUF configurations can provide another path without that module. A successful local GPU peer test does not verify GPU-to-NIC direct communication.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstall3. Test the InfiniBand fabric and identify the rails
Check that the intended compute-side InfiniBand interfaces are up, then measure the inter-node fabric independently of the NCCL collective. NVIDIA’s NCCL diagnostics can run ib_write_bw over physical InfiniBand devices selected by NCCL when the communicator spans at least two hosts. The ib_write_bw utility from perftest must be available on every participating node, and the hostnames must resolve between them. NVIDIA’s NCCL 2.31.2 performance guidance also identifies ib_write_lat for fabric latency checks.
When the endpoints and installed tool support it, the bandwidth test uses GPU memory; otherwise it falls back to host memory. Record which memory path was measured. A host-memory result is useful for assessing the fabric, but it is not proof that GPU-direct transfers are working.
Rank #2
- Professional GPU with Blackwell Architecture
- Blackwell Architecture
- 24GB GDDR7 with PCIe 5.0 & Ray Tracing
- AI Workstation
Interpret same-NIC and cross-NIC measurements against your topology
NCCL diagnostics report same-NIC and cross-NIC modes separately when those checks are scheduled. Preserve the device identity for each result and map each NIC to its rail, GPU locality, and node. The diagnostic follows devices selected according to NCCL’s topology and NCCL_CROSS_NIC behavior; it does not necessarily exercise every physical path in the cluster.
“Cross-NIC” describes how NCCL uses network interfaces in the tested configuration. It should not automatically be read as proof that traffic traversed every intended physical rail. Compare the reported modes with the actual NIC-to-rail mapping and the deployed value of NCCL_CROSS_NIC; do not change that setting without measuring the effect on the target workload.
4. Run a multi-node NCCL correctness and performance test
After the local and fabric checks, run a supported multi-node NCCL test job on the intended nodes, GPUs, and network resources. NVIDIA’s Kubernetes validation example uses NCCL tests over high-speed links and lists MPI Operator, GPU Operator, and Network Operator as prerequisites. Use a job template supported by your distribution and operator versions rather than copying a manifest from a different environment.
Check correctness before treating timing as useful: a fast result is not valid if the collective fails or produces incorrect output. Then collect results across message sizes and collective operations relevant to the application. Record the rank-to-node and rank-to-GPU placement alongside the output; otherwise a rank imbalance can be hard to distinguish from an uneven rail or device path.
DCGM’s NCCL Tests plugin is not a substitute for this cross-node test. NVIDIA documents that the plugin runs only single-node NCCL tests and does not support multi-node tests. It can still help check local NCCL behavior when the required NCCL library, test binary, and executable path are installed and configured.
Rank #3
- NVIDIA Ampere Architecture-based CUDA Cores - Double-speed processing for single-precision floating point (FP32) operations and improved power efficiency provide significant performance improvements for graphics and simulation workflows, such as complex 3D computer-aided design (CAD) and computer-aided engineering (CAE), on the desktop.
- Second-Generation RT Cores - With up to 2X the throughput over the previous generation and the ability to concurrently run ray tracing with either shading or denoising capabilities, second-generation RT Cores deliver massive speedups for workloads like photorealistic rendering of movie content, architectural design evaluations, and virtual prototyping of product designs. This technology also speeds up the rendering of ray-traced motion blur for faster results with greater visual accuracy.
- Third-Generation Tensor Cores - New Tensor Float 32 (TF32) precision provides up to 5X the training throughput over the previous generation to accelerate AI and data science model training without requiring any code changes. Hardware support for structural sparsity doubles the throughput for inferencing. Tensor Cores also bring AI to graphics with capabilities like DLSS, AI denoising, and enhanced editing for select applications.
- Third-Generation NVIDIA NVLink - Increased GPU-to-GPU interconnect bandwidth provides a single scalable memory to accelerate graphics and compute workloads and tackle larger datasets.
- 48 Gigabytes (GB) of GPU Memory - Ultra-fast GDDR6 memory, scalable up to 96 GB with NVLink, gives data scientists, engineers, and creative professionals the large memory necessary to work with massive datasets and workloads like data science and simulation.
5. Read the diagnostics and isolate the slow layer
Separate a connectivity warning from a performance target
In NCCL diagnostics, [OK] means a check completed without reporting an issue; [INFO] means a condition merits review, such as failed verification or a check that could not be completed. A passing peer-to-peer check makes an intra-node or NVLink connectivity issue less likely, but it does not prove the inter-node network or application is healthy.
The diagnostics report per-rank minimum, median, and maximum bandwidth for same-NIC and cross-NIC modes. They can flag a rank whose result is more than 30% away from that mode’s median. That is the diagnostic’s outlier-reporting rule, not a universal bandwidth acceptance threshold or cluster service-level objective. Likewise, the documentation’s illustrative output in which 52 of 56 GPU-to-GPU peer accesses passed is an example of partial results, not a recommended pass count.
Use the pattern of results to choose the next check
- Local GPU bandwidth or peer verification is poor: investigate GPU topology, peer access, drivers, and the relevant local paths before changing network tuning.
- Local checks pass but fabric measurements are poor: verify interface state, host resolution, NIC identity, rail mapping, and whether the tested memory path used GPU or host memory.
- Standalone local and fabric measurements meet the deployed hardware’s expected performance, but NCCL is slow: inspect job placement and NCCL configuration. NVIDIA’s performance guidance names
NCCL_CROSS_NIC, queue pairs per connection, chunk sizing, and CPU/memory affinity as variables that can affect results. - Only certain ranks or one NIC mode are slow: compare the outlier ranks’ node, GPU, NIC, and rail assignments rather than relying only on an overall average.
NCCL tuning is system- and workload-specific. Change one variable at a time and compare under the same placement, collective, and message-size mix: a setting that helps one benchmark can be suboptimal for another.
What counts as a useful baseline
There is no single cross-rail bandwidth number that can validate an unspecified Kubernetes GPU cluster. The meaningful reference depends on the GPU and NIC generations, number of rails, topology, drivers, memory path, collective, message size, and workload. Compare each measurement with the expected performance for the hardware and path actually deployed, and keep the standalone GPU, fabric, and end-to-end NCCL results separate.
NVIDIA’s referenced NCCL diagnostics page is for NCCL 2.32.3, while the cited performance and GPU-troubleshooting guidance is from NCCL 2.31.2. Check version-specific instructions against the NCCL, driver, and operator versions installed in your cluster.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
