Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Keep inference on one GPU or one host whenever the model fits. Use tensor parallelism inside a multi-GPU host, then add pipeline parallelism between hosts when the model exceeds one machine. Ordinary private Ethernet can work for llama.cpp RPC and some pipeline-parallel designs, but cross-node tensor parallelism usually needs a carefully matched InfiniBand or RoCE/RDMA fabric. The right choice depends on model size, KV-cache and concurrency requirements, GPU topology, software stack, and measured bandwidth and latency—not on link speed alone.
What network setup should you choose first?
Start by deciding whether distribution is necessary at all. Distributed inference adds communication, synchronization, security controls, storage coordination and another class of failure. If the model and its expected KV-cache fit on one GPU, keep it there. If it fits in one server with several GPUs, use that server before adding a second host.
As an Amazon Associate I earn from qualifying purchases.
Recommended Free Tools
- One GPU is sufficient: use no distributed network.
- Several GPUs in one host are sufficient: use intra-node tensor parallelism and the host’s GPU interconnect.
- The model exceeds one host: combine tensor parallelism within each host with pipeline parallelism between hosts.
- You are extending llama.cpp to another local machine: start with private wired Ethernet and TCP, then measure whether the link is limiting generation.
This progression is the common guidance for vLLM and NVIDIA-style multi-node deployments. It avoids paying for a high-speed fabric when the workload does not need one, while preserving a path to scale when a model cannot be placed on a single machine.
How do tensor and pipeline parallelism change the network requirement?
Tensor parallelism splits operations across GPUs that must cooperate during inference. Pipeline parallelism assigns different groups of layers to different stages. That difference determines how continuously the network is used.
#1 Best Overall
- 𝐅𝐢𝐯𝐞 𝟏𝟎𝐆𝐛𝐩𝐬 𝐏𝐨𝐫𝐭𝐬 𝐟𝐨𝐫 𝐋𝐢𝐠𝐡𝐭𝐧𝐢𝐧𝐠-𝐅𝐚𝐬𝐭 𝐂𝐨𝐧𝐧𝐞𝐜𝐭𝐢𝐨𝐧𝐬: 5× 10-Gigabit ports unlock the highest performance with 10G/multi-gig bandwidth and provide up to 100 Gbps of switching capacity.
- 𝐀𝐮𝐭𝐨-𝐍𝐞𝐠𝐨𝐭𝐢𝐚𝐭𝐢𝐨𝐧: Auto-negotiation intelligently senses the link speeds and adjusts between 5-speeds (100Mb/1G/2.5G/5G/10G) for compatibility and optimal performance for all your devices, including 2.5G/5G/10G WiFi 6 AP, 10G NAS, 10G PCIe Adapter/NIC, 10G Server, gaming computer, 8K video, and more.
- 𝐑𝐞𝐥𝐢𝐚𝐛𝐥𝐞 𝐚𝐧𝐝 𝐐𝐮𝐢𝐞𝐭: IEEE 802.3X flow control provides reliable data transfer and a fanless design ensures quiet operation.
- 𝐏𝐥𝐮𝐠 𝐚𝐧𝐝 𝐏𝐥𝐚𝐲: Easy setup with no software installation or configuration needed.
- 𝐒𝐭𝐮𝐫𝐝𝐲 𝐌𝐞𝐭𝐚𝐥 𝐂𝐚𝐬𝐞: Durable metal casing and desktop/wall-mounting design are well-suited for different environments.
| Approach | Where work is placed | Communication pattern | Typical network implication |
|---|---|---|---|
| Intra-node tensor parallelism | Multiple GPUs in one server | Frequent exchanges over the host’s GPU interconnect | Prefer keeping the tensor-parallel group inside one host. |
| Cross-node tensor parallelism | Tensor shards span hosts | Continuous GPU-to-GPU communication across nodes | Bandwidth, latency and RDMA availability directly affect performance; InfiniBand or RoCE/RDMA is preferred. |
| Pipeline parallelism | Layer ranges are assigned to stages, often one stage per host | Activations move between stages rather than every tensor operation crossing hosts | High-speed Ethernet may be adequate if measured latency and throughput meet the workload. |
| llama.cpp RPC offload | A main process uses remote worker processes | RPC traffic over TCP by default, with optional negotiated RDMA when supported by both peers | Private wired Ethernet is the baseline; local caching avoids repeatedly moving large tensors. |
For a two-host design, a practical pattern is to keep each host’s tensor-parallel group local and use pipeline parallelism between hosts. This reduces continuous cross-node tensor traffic, but it does not eliminate the need for a reliable, low-latency link.
Is 10GbE enough for multi-GPU inference?
There is no universal yes-or-no answer. A 10GbE link can be a reasonable starting point for llama.cpp RPC on a private LAN or for a pipeline-parallel design whose measured activation traffic fits the link. It is a risky default for cross-node tensor parallelism, where every generation step can depend on repeated synchronization across the network.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11When 10GbE is a reasonable choice
- The model fits across the chosen hosts using pipeline stages rather than cross-node tensor shards.
- Remote workers are used mainly for capacity and the local RPC cache prevents repeated transfer of large, mostly static tensors.
- Your target concurrency is modest and a pilot run shows acceptable token latency and throughput.
- The link is dedicated or isolated from storage, backup and general user traffic.
When to plan for a faster fabric
- Tensor-parallel groups must span hosts.
- Several concurrent requests create sustained synchronization traffic.
- NCCL reports socket transport rather than RDMA transport in a configuration expected to use a high-speed fabric.
- Increasing GPUs produces little or no throughput gain because communication, rather than compute, is the bottleneck.
Do not infer suitability from the port label alone. Measure the intended model, quantization, context length and concurrency on the actual hosts. Official guidance does not provide a single safe link-speed number, and workload and topology can change the result substantially.
Do you need InfiniBand or RDMA for llama.cpp?
No. llama.cpp RPC can use TCP, so ordinary private wired Ethernet is the baseline for a local deployment. The RPC implementation can negotiate RDMA when both peers support it, but the feature and command-line options are version-sensitive and the RPC backend is explicitly described as proof-of-concept software.
TCP is the simplest starting point
Use a switched, wired network rather than Wi-Fi, place the machines on a private subnet or VLAN, and keep the RPC ports reachable only from trusted worker and controller addresses. Enable the RPC local cache so large tensors are not transferred again whenever the same data can be retained on the worker.
RDMA is an optimization, not a prerequisite
InfiniBand and RoCE provide the low-latency, high-throughput paths commonly preferred for NCCL-heavy multi-node tensor parallelism. They require compatible NICs, switch support, cabling, drivers and software configuration on every participating host. Adding an RDMA NIC without validating the complete path can leave the application using ordinary sockets.
The llama.cpp RPC README warns that the example and RPC backend are fragile and insecure and should never be exposed on an open network or used in a sensitive environment. Treat that warning as an architectural requirement, not as a tuning note.
What hardware must match across the hosts?
A network purchase is only one part of the design. The usable configuration is constrained by the GPUs, host slots, PCIe topology, drivers, switch ports and cable runs.
GPU and PCIe topology
Check which GPUs share PCIe root complexes, whether the host has a high-bandwidth GPU interconnect, and which slot each NIC will occupy. A NIC placed behind a congested or remote PCIe path can reduce the benefit of an otherwise capable adapter. Keep the tensor-parallel group aligned with the fastest local GPU paths whenever the software permits it.
NIC and switch compatibility
For RDMA, match the NIC generation and supported protocol with the switch and the operating-system drivers. RoCE deployments also need a correctly configured lossless or near-lossless Ethernet environment; a nominally compatible switch is not automatically a working RDMA fabric. Count ports for every host, management needs and any separate storage network.
Cables and distance
Choose copper or optical cabling according to the adapter and switch requirements and the distance between racks. Confirm transceiver compatibility before installation. A link that negotiates at a lower rate, flaps under load or shares a congested uplink can invalidate performance assumptions.
Rank #2
- ⭐【8 Port 10Gbe Ethernet Switch】8*10G/5G/2.5G/1000M/100M RJ45 Ethernet ports, 160Gbps switching capacity to meet the broadband needs of more devices, build more stable and larger network.
- ⭐【Auto-Negotiation】Autonegotiation intelligently senses link speed and adjusts between 5 speeds (10G/5G/2.5G/1000M/100M) to provide compatibility and optimal performance for all your devices, including 10G NAS, WiFi7 Router,10G Adapter/NIC, Server, Gaming PC, 8K video, etc.
- ⭐【Plug and Play】Easy to use and simple to set up, no software or configuration required. Supports automatic MDI/MDIX and non-blocking data forwarding.Support Desktop/Rackmount(with 19-inch rack ear)
- ⭐【Widely Used】Durable Metal Case, 4KV Lightning Protection, Industrial Grade 5000 RPM Fan (24dB), Dual Side Cooling Holes, Qperating Temperature 0°C to 45°C (32°F to 113F), LED Indicator Lights, Stable Operation, Easily Maintain the Network Status.
- ⭐【After-Sale-Service】 Every NICGIGA switch is rigorously tested for reliability, quality and performance. We provide a one-year warranty and lifetime technical support for the entire product.
Storage placement
Keep model files on every host or provide dependable shared storage. Local copies avoid startup dependence on a storage network; shared storage simplifies updates but introduces another latency and availability path. Whichever method you choose, every process must see the same model revision, tokenizer files and runtime configuration.
What software prerequisites should be identical?
Use the same execution environment on all nodes. Match the operating-system family, GPU driver, CUDA or other accelerator runtime, framework version, communication library and model files. Small differences can cause a worker to fail during initialization or force a slower transport.
- Install the same inference framework and version on every node.
- Install matching GPU drivers and accelerator libraries.
- Use identical model and tokenizer files, including quantization and revision.
- Synchronize clocks enough for useful logs and operational diagnosis.
- Reserve stable hostnames or addresses for controller and worker roles.
- Document which interface carries inference traffic when hosts have multiple NICs.
For vLLM or NVIDIA-style deployments, configure the cluster manager, Ray components and NCCL for the selected topology. Pre-stage model files or mount shared storage before starting the workers; a node that can join the cluster but cannot read the model is not ready to serve requests.
Free tools Windows power users keep installed
One-click scans. No signup required.
How do you connect two machines with llama.cpp RPC?
Use the version-specific build and help output because RPC options and transport support can change. The sequence below is the safe operational pattern.
- Build RPC support on both machines. Build the worker binary and the main inference binary from the same llama.cpp revision. Confirm the resulting binaries expose RPC-related options in their help text.
- Assign private addresses. Put the hosts on a private switch or VLAN. Give each worker a stable address and decide which TCP port the RPC server will listen on.
- Start each worker on its private interface. Bind the RPC server to the address reachable only by the controller and firewall the listening port so that only approved node addresses can connect.
- Start the main process with worker endpoints. Supply the worker host-and-port endpoints through the version’s
--rpcoption. A representative form is--rpc 10.0.0.12:PORT,10.0.0.13:PORT; verify the exact separator and address syntax with--helpfor your build. - Enable the RPC local cache. Use the cache option provided by your build so large tensors already present on a worker are not repeatedly sent over the link.
- Run a small validation prompt. Confirm that all workers connect, the model loads, and generation completes before testing long context or multiple concurrent requests.
- Record transport and errors. Save startup logs, connection failures, memory allocation messages and per-request latency. Repeat the test after any driver, model or network change.
Do not publish the RPC port to the internet. The proof-of-concept status means authentication, hardening and failure behavior may not meet production expectations even when the model output is correct.
How should you configure vLLM or an NVIDIA multi-node stack?
Set tensor-parallel size to the number of GPUs in each node and pipeline-parallel size to the number of nodes when that matches the hardware plan. For example, two nodes with eight GPUs each commonly map to tensor parallelism of eight and pipeline parallelism of two. This is a deployment pattern, not a guarantee that every model or scheduler will scale efficiently.
- Install identical framework, driver, accelerator and NCCL environments.
- Make the model and tokenizer available locally or through a tested shared-storage path on every node.
- Define the node inventory and ranks in the cluster manager.
- Set tensor and pipeline parallel sizes to match the physical topology.
- Bind communication to the intended NICs and configure the RDMA or Ethernet fabric according to the vendor and framework documentation.
- Launch a single replica and inspect logs before adding replicas or higher concurrency.
- Confirm the selected transport in NCCL logs:
NET/IB/GDRDMAindicates an RDMA path, whileNET/Socketindicates raw TCP sockets.
If a deployment intended to use RDMA reports socket transport, investigate interface selection, drivers, permissions, cable and switch configuration before interpreting performance results.
How should you secure a distributed inference network?
Separate inference traffic from the public internet. Use a private switch or VLAN, host firewalls and allowlists that admit only the controller and known workers on the required ports.
- Do not bind experimental RPC services to a public address.
- Restrict security-group or firewall rules to specific node addresses, not an entire internet-facing range.
- Keep management interfaces separate from the data path where practical.
- Log connection attempts and failed worker authentication or handshakes.
- Remove temporary test exposure after validation.
- Assume that anyone who can reach an unauthenticated experimental RPC endpoint may be able to disrupt the service or influence work performed by the worker.
Isolation does not replace application security. Review the exact release notes and configuration reference for the inference software before placing it in a shared research or production environment.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How do you validate that the network is helping rather than hurting?
Benchmark the complete workload, not an isolated link test. Use the target model, quantization, context lengths, request mix and concurrency on the final hardware.
Measure these outcomes
- Time to first token and steady-state tokens per second.
- Latency at one request and at the target concurrent request count.
- GPU utilization and memory pressure on every node.
- Network throughput, retransmissions, errors and link negotiation rate.
- Startup time, model-transfer time and cache-hit behavior.
- Failure recovery when a worker restarts or becomes unreachable.
Use logs to identify the transport
NCCL logs distinguish NET/IB/GDRDMA from NET/Socket. A socket result is not automatically wrong for pipeline parallelism or llama.cpp RPC, but it is a warning when you expected an RDMA-enabled tensor-parallel deployment.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesRank #3
- 【10G Ethernet Switch】The network switch includes 16 x 2.5Gb Ethernet ports, 2 x 10Gb RJ45 Port.Auto MDI/MDIX.1-16 Port Support 2.5G/1G/100Mbps, 17-18 Port Support 10G/5G/2.5G/1G/100Mbps. Provide compatibility and optimal performance for all your devices, including 10G NAS, WiFi7 routers, 10G adapters/NICs, servers, gaming PCs, 8K video, etc. Build a more stable and larger network.
- 【Super Fast Network Speeds】This network switch delivers 2.5x faster network performance than traditional Gigabit switches. With a switching capacity of up to 120Gbps, this network switch can handle large amounts of data traffic, ensuring smooth and efficient network operation.
- 【Plug and Play】Easy to use and simple to set up, no software or configuration required. Supports automatic MDI/MDIX and non-blocking data forwarding.Support Desktop/Wall Mount.
- 【Stable and Quiet】Durable metal case with 6KV lightning protection, fanless silent design for more power saving, double side cooling holes, wide temperature range -10~50°C, LED indicator lights, stable operation, easy to maintain network status.
- 【After-Sale-Service】 Every NICGIGA switch is rigorously tested for reliability, quality and performance. We provide a one-year warranty and lifetime technical support for the entire product.
Compare a local baseline
Run the same model on one host when possible, then compare the distributed configuration at the same context and concurrency. If adding a node lowers throughput or sharply increases token latency, inspect communication volume, rank placement, NIC selection and synchronization before buying more hardware.
What common failures should you troubleshoot first?
Workers cannot connect
Check private-address routing, host firewalls, listening interfaces and the exact port. Confirm that the worker is listening on the address the controller is using, rather than only on loopback or a management interface.
The model loads on one node but not another
Compare model paths, file checksums, tokenizer files, available disk space, GPU memory and runtime versions. A shared path can exist while permissions or mount timing still prevent a worker from reading it.
Performance is much worse than expected
Inspect NCCL transport logs, GPU-to-NIC topology, negotiated link rate and switch errors. Cross-node tensor parallelism over raw sockets is often a sign that the intended RDMA path is unavailable or misconfigured.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Generation fails only at longer context
Recalculate KV-cache capacity and per-rank memory. A model that fits at a short prompt can exhaust memory or trigger additional communication at the target context length and concurrency.
Repeated transfers dominate startup or requests
Verify that the llama.cpp RPC local cache is enabled and persistent for the worker process. For framework-based serving, pre-stage model files or use a tested shared-storage design rather than copying large artifacts during every launch.
Adding a second host does not improve throughput
Check whether the workload is compute-bound on the original host, whether the parallelism choice matches the model, and whether synchronization traffic saturates the link. More nodes increase capacity only when the model partition and communication path can use them efficiently.
Which architecture fits common scenarios?
| Scenario | Recommended starting design | Main trade-off |
|---|---|---|
| 7B-class or otherwise single-GPU model | One GPU, no distributed network | Lowest complexity and latency; limited by that GPU’s memory and throughput. |
| Large model that fits in one multi-GPU server | Intra-node tensor parallelism | Requires a suitable GPU topology but avoids cross-host synchronization. |
| llama.cpp across two ordinary local machines | Private wired Ethernet, TCP RPC and local cache | Simple and inexpensive to deploy, but RPC is experimental and may become bandwidth-bound. |
| Model larger than one host, moderate network capability | Tensor parallelism within hosts plus pipeline parallelism between hosts | Less cross-node traffic, but stage balance and inter-stage latency affect utilization. |
| High-concurrency cross-node tensor parallelism | Matched InfiniBand or RoCE/RDMA NICs and switch, with NCCL validation | Higher hardware and operational complexity; required performance depends on workload. |
What should you decide before buying a switch or NIC?
- Measure model weights, KV-cache growth and target concurrency.
- Determine the minimum number of GPUs and hosts that can meet capacity goals.
- Choose tensor, pipeline or combined parallelism based on where communication occurs.
- Map GPU, PCIe, NIC and switch topology on every host.
- Decide whether private TCP is sufficient for a pilot or whether the workload requires RDMA.
- Confirm cable distance, switch ports, driver support and software transport support.
- Stage the model and run a representative benchmark before expanding the cluster.
The least expensive network is the one that meets the measured workload without forcing a communication bottleneck. A more expensive fabric is justified when cross-node tensor traffic is unavoidable and the benchmark shows that bandwidth or latency is limiting useful work.
Frequently Asked Questions
Can Wi-Fi be used for distributed local AI inference?
It may work for a noncritical experiment, but it is a poor choice for sustained RPC or parallel inference. Variable latency, interference and roaming make timing less predictable than a dedicated wired path, especially when requests synchronize between hosts.
Should model files be copied to every node or stored on a NAS?
Local copies remove storage-network latency and reduce startup dependencies, while shared storage simplifies version management. Choose one deliberately and verify that every worker can read the same model revision before launching distributed inference.
How many GPUs should belong to one tensor-parallel group?
Keep the group within a host whenever possible, using the number of GPUs that share the fastest local topology. Span hosts only when capacity requires it and the network has been measured for the resulting communication pattern.
Why does a distributed model use more memory than expected?
Each process may need runtime buffers, communication workspaces and a portion of the KV-cache in addition to model weights. Context length, concurrency, quantization and the selected parallelism determine the real per-GPU requirement.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
