Multi-GPU inference is not a single-GPU run repeated several times. The model may be split across devices, replicated as separate workers, or processed through pipeline stages. Those arrangements add communication and synchronization, so extra GPUs can increase model capacity or total throughput without proportionally reducing the latency of one request. To compare results fairly, match the model, precision, workload, metric, parallelism strategy, GPU topology, software stack and measurement method.
Capacity, latency and throughput are different outcomes
The first question is what “faster” means. A multi-GPU system can solve a capacity problem while leaving single-request latency unchanged, or it can serve more requests per second while each request takes about as long. Treating these outcomes as interchangeable is the most common reason benchmark claims appear to conflict.
As an Amazon Associate I earn from qualifying purchases.
Capacity
Sharding a model across GPUs lets its weights and working state fit when one device lacks sufficient memory. This is often the reason to use tensor or pipeline parallelism. A model that cannot load on one GPU has no single-GPU latency result to scale from; the relevant comparison is whether it runs at all and what service level it achieves on the multi-GPU system.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Single-request latency
Latency can be reported as time to first token (TTFT), decode or per-token latency, or complete end-to-end time. A request divided among GPUs must often wait for intermediate data and synchronization before the next layer or token can proceed. Consequently, two GPUs may lower neither TTFT nor total completion time, particularly for a small batch or a short request.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
Aggregate throughput
Throughput measures work completed over time, such as total tokens per second across concurrent requests. More GPUs can raise this number by providing more parallel work, larger batches or additional model replicas even when the latency of an individual request does not improve. A throughput result therefore cannot be used as a promise about interactive response time.
How the parallelism strategy changes the result
Tensor parallelism
Tensor parallelism divides operations within layers among GPUs. Each rank computes part of a layer and exchanges intermediate results with the other ranks. vLLM’s current stable guidance recommends tensor parallelism when a model is too large for one GPU but fits within a single multi-GPU node. That is a framework recommendation for that situation, not a universal rule for every model or machine.
TensorRT-LLM describes a representative synchronization point: GPUs may need the complete output of a preceding layer, so the ranks perform an all-reduce before continuing. The collective operation introduces a barrier; a fast rank can wait for a slower rank or for data to cross a link. The amount of computation saved by splitting a layer must therefore exceed the time spent moving and synchronizing its results.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePipeline parallelism
Pipeline parallelism assigns groups of layers to different devices. Activations move between stages rather than every layer being divided in the same way. Depending on the hardware and workload, this can reduce communication relative to tensor parallelism. vLLM notes that certain hardware configurations can obtain higher throughput and lower communication overhead with pipeline parallelism.
Pipeline execution also introduces stage scheduling and “bubbles”: a stage can sit idle while it waits for work from the preceding stage or for a later stage to become available. A larger batch or multiple in-flight requests can fill those gaps, while a single request may expose them.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Data parallelism and replicas
Data parallelism keeps a complete model copy on each group of GPUs and sends different requests to different copies. Additional GPUs then increase capacity and aggregate serving throughput rather than accelerating one request. When a benchmark says it used four GPUs, determine whether they formed one sharded model instance or several replicas; the two arrangements answer different performance questions.
Communication, topology and synchronization
GPU count alone does not describe the hardware path. Intra-node links, PCIe arrangements, NVLink or NVSwitch fabrics, and the network used between nodes determine how quickly ranks exchange tensors. A multi-node deployment adds network latency and bandwidth limits that do not exist when all GPUs share one node.
Free tools Windows power users keep installed
One-click scans. No signup required.
NVIDIA’s technical discussion of multi-GPU large-language-model inference explains that time spent exchanging results can leave Tensor Cores idle while they wait for data. Its examples use NVLink and NVSwitch, so the reported behavior should not be generalized to a fixed speedup on other systems.
One concrete example is configuration-specific: NVIDIA reports up to 20 GB of tensor-parallel synchronization data per GPU for one Llama 3.1 70B query with an 8K-token input and a 256-token output. That figure describes the stated model, token lengths and parallel execution; it is not a universal traffic amount for every query.
Synchronization also creates a “slowest participant” effect. If one GPU has less available bandwidth, shares a bus with another device, or runs at a lower clock because of thermal or power limits, the other ranks can wait at collectives. A benchmark that changes the interconnect, node count or GPU placement is not measuring GPU count in isolation.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Why batch size and concurrency change the winner
Batching determines how much independent work is available to amortize fixed costs. Larger batches can keep more execution units busy and improve aggregate tokens per second. They also consume more memory and generally increase queueing or completion latency. Continuous batching in a serving system further changes the request mix during the run, so its result is not directly comparable with a fixed offline batch.
Low-batch workloads
For one request or a very small batch, communication, synchronization and kernel-launch overhead can be a large fraction of total time. Triton’s TensorRT multi-device documentation describes replicated rank-0 inputs, collective operations and extra launches as costs that can make a low-batch, single-attention-layer case neutral-to-modest and noisy on multiple GPUs. This is a documented example, not a statement about all models.
High-throughput workloads
At higher concurrency, the same communication can be amortized over more computation. A multi-GPU setup may then deliver substantially greater total throughput, especially when it enables a larger effective batch or keeps several pipeline stages busy. The improvement still depends on memory limits, request lengths and the communication fabric.
Metrics that must not be mixed
| Metric | What it answers | Why it differs across setups |
|---|---|---|
| Time to first token (TTFT) | How quickly generation begins after a prompt arrives | Prompt processing, input length, batching and synchronization dominate; it is not decode speed. |
| Decode latency or tokens per second per request | How quickly one active request generates tokens | Per-token collectives and pipeline scheduling can offset split computation. |
| End-to-end latency | Elapsed time from request submission to the final token | Combines queueing, prompt work, decode, output length and runtime overhead. |
| Aggregate tokens per second | Total serving capacity across requests | Benefits from batching, concurrency and replicas; it does not imply lower per-request latency. |
Always state the metric’s unit, whether it is a mean, median or tail percentile, and the measurement window. A throughput number from an offline batch and a TTFT number from a live service are not competing measurements.
The comparison checklist for conflicting benchmarks
When two published results disagree, compare these fields before blaming the GPU count:
Recommended Free Tools
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
| Axis | Details to record |
|---|---|
| Workload | Exact model and version, architecture or parameterization, precision or quantization, input-token length, generated-output length and request mix. |
| Metric | TTFT, per-token decode, end-to-end latency, or aggregate tokens per second; include percentile and measurement window where available. |
| Load | Batch size, concurrency, request arrival pattern, and static versus continuous batching. |
| Parallelism | Tensor, pipeline, data or hybrid parallelism; GPUs per model instance; and whether extra GPUs are replicas or shards. |
| Hardware path | GPU model and memory, GPU count, node count, PCIe/NVLink/NVSwitch arrangement, and the multi-node network if used. |
| Software and method | Inference framework and version, CUDA and NCCL (or equivalent communication stack), warm-up procedure, run duration, prompt generation and offline versus live serving. |
These fields form a practical reporting checklist, not a single cross-framework standard. If a result omits several of them, its number should be treated as conditional rather than portable.
How to run an apples-to-apples comparison
- Fix the model. Use the same checkpoint, tokenizer, model revision and quantization on every configuration. Record any changes to maximum sequence length or attention implementation.
- Fix the request shape. Choose explicit input and output lengths, or publish a distribution for them. Keep the prompt set and generation settings identical.
- Define the question. Decide whether you need single-request latency, tail latency under a service load, or maximum aggregate throughput. Do not optimize one metric and report it as another.
- Pin the parallel layout. Document tensor, pipeline or data parallelism, the number of GPUs per instance, replica count and node placement.
- Record the hardware path. List GPU type and memory, interconnect, topology and node count. A two-node run should not be labeled equivalent to a two-GPU single-node run without qualification.
- Warm up before timing. Run enough requests to load weights, initialize kernels and establish the communication context. Exclude startup unless startup time is the subject of the test.
- Control load and batching. Test the stated batch or concurrency levels and keep request arrival behavior consistent. For a serving test, report whether batching is static or continuous.
- Capture distributions. Publish sample count, duration and latency percentiles as well as averages. Short runs can be dominated by initialization, scheduling or background activity.
- Repeat and inspect outliers. Verify that all ranks remain healthy and that clocks, power limits and thermal conditions are comparable. Investigate noisy runs instead of selecting only the fastest repetition.
Reading scaling numbers correctly
Speedup is commonly expressed as S(n) = T(1) / T(n), where T is the same metric at one and n GPUs. Scaling efficiency is E(n) = S(n) / n. These calculations are meaningful only when the workload, metric and execution mode remain fixed. If the one-GPU test uses batch one and the four-GPU test uses batch 16, the calculation mixes hardware scaling with a workload change.
Sublinear speedup is expected when communication and synchronization grow with the number of ranks, when part of the model remains serial, or when fixed launch and input-replication costs dominate. A result below one can also be valid: adding GPUs may make a small workload slower while providing the memory capacity needed for a larger model.
Common reasons a multi-GPU result looks disappointing
The model was replicated instead of sharded
Replication improves concurrent capacity, not the latency of one request. Check the serving topology and the number of requests in flight.
The workload is too small
Batch-one or short-sequence tests expose collective and launch overhead. Increase concurrency only if that reflects the intended service, and report the resulting latency trade-off.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
The interconnect is the bottleneck
Confirm device placement, link type, peer-to-peer access and whether traffic crosses a host bus or node boundary. A slower path can erase the arithmetic advantage of splitting layers.
The metric changed
Check that one result is not TTFT while another is end-to-end latency or aggregate throughput. Also verify that output length and queueing behavior match.
The software stack changed
Framework, CUDA, NCCL, kernel, precision and communication settings can alter launch order and collective performance. Pin versions and configuration before drawing a hardware conclusion.
A practical decision framework
- Need the model to fit? Choose a sharding strategy and prioritize memory capacity and a suitable interconnect; speedup is secondary.
- Need lower interactive latency? Benchmark TTFT and end-to-end latency at the actual request shape and concurrency. Do not substitute aggregate throughput.
- Need more requests per minute? Measure sustained aggregate tokens per second with realistic batching, replicas and arrival rates.
- Choosing between tensor and pipeline parallelism? Compare both on the target topology. Tensor parallelism can offer the required fit within one node, while pipeline parallelism may reduce communication overhead on particular hardware.
- Comparing published numbers? Reject any apparent “GPU scaling” claim until model, precision, metric, load, topology, parallelism and software details align.
What a trustworthy benchmark report contains
A useful report lets another engineer reproduce the question being answered. It names the exact model and precision, token lengths, batch or concurrency, metric definition, GPU and node configuration, interconnect, parallelism mode, framework and communication-stack versions, warm-up policy, run duration and statistical summary. It also separates capacity, single-request latency and aggregate throughput instead of presenting one as a proxy for the others.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
