Halloween DealsAmazon USHalloween week deals: useful picks before month-endAmazon US: current deals, useful picks and tech finds.Check DealsWindows FixRecommendedWindows errors stealing your time? Find the fix fastScan stability, cleanup and performance issues.Fix NowHalloween DealsAmazon USOctober picks are moving - compare before checkoutAmazon US: current deals, useful picks and tech finds.See Picks×
FREEDOM251

Freedom251 › Blog › AI networking

Scale-Up vs. Scale-Out Networking: When to Use Ultra Ethernet 1.1

By Freedom251 Team · · 10 min read

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Use scale-up networking inside a tightly coupled accelerator pod, scale-out networking between pods and racks, and both in a large AI cluster. Ultra Ethernet Specification 1.1 (UE 1.1), released by the Ultra Ethernet Consortium on October 12, 2026, is designed to cover that two-level reality. Its new Unified Local profile, smaller forwarding overhead, collective-operation offload, and programmable congestion controls target scale-up traffic while preserving the routing and interoperability expected of Ethernet scale-out fabrics.

Scale-up and scale-out solve different networking problems

The distinction is about communication locality, not simply the number of switches. In a distributed-training job, accelerators may exchange tensors thousands of times per second with neighbors in the same chassis or pod, while the same job also moves data between pods, racks, storage systems, and service networks. Those paths have different performance bottlenecks and failure boundaries.

As an Amazon Associate I earn from qualifying purchases.

Characteristic Scale-up networking Scale-out networking
Primary scope Accelerators inside one tightly coupled pod or domain Many servers, pods, racks, or sites in one fabric
Main objective Very low latency, high bandwidth density, low packet overhead, and efficient collectives Reachability, routing, congestion control, multipathing, fault isolation, and predictable utilization
Traffic pattern Frequent reads, writes, atomics, and synchronized all-reduce or all-to-all operations Bursty distributed-training flows, incast, storage traffic, and many simultaneous jobs
Failure concern Keeping a small accelerator domain tightly synchronized Containing faults and congestion so one link, rack, tenant, or job does not disrupt the fabric
Typical design question Can every accelerator exchange data quickly enough for the next training step? Can thousands of endpoints sustain traffic while paths change and jobs compete?

A scale-up link can be excellent within a pod yet unsuitable as the only network for a multi-rack installation. Conversely, a fabric optimized for broad Ethernet reach can waste bandwidth and add latency when every operation is between neighboring accelerators. Modern AI factories therefore commonly use a scale-up layer and a scale-out layer, with gateways or network interfaces joining them.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

What Ultra Ethernet 1.1 changes

UE 1.1 extends the Ultra Ethernet stack beyond a generic Ethernet data path. The specification addresses transport behavior, congestion signaling, collective operations, forwarding, and interoperability for AI and high-performance-computing fabrics. Its most important change for this decision is explicit support for scale-up endpoints through the Unified Local profile.

#1 Best Overall
Mikrotik - CRS520-4XS-16XQ-RM 2 2.5Gb 4 SFP28 25Gb 16 WSFP28 100Gb Switch
  • Ultra-fast enterprise switch
  • Extra processing power due to the CCR series CPU
  • 28 Gigabit SFP28 ports
  • 100 Gigabit QSFP28 ports
  • 10 Gigabit Ethernet ports

Smaller packets and a Unified Forwarding Header

UEC states that reducing the inter-packet gap can save as many as 11 idle bytes. That saving matters when a stream consists of many small messages: the wire spends less time carrying spacing rather than payload, and packet-rate pressure on adapters and switches falls.

The Unified Forwarding Header (UFH) saves 20 bytes compared with a traditional IP packet while retaining IP routing and Ethernet interoperability. The result is not a new isolated transport island. An implementation can use the lower-overhead UE format for the traffic that needs it while continuing to operate in routed Ethernet environments. The exact benefit depends on packet size, headers selected by the implementation, link rate, and how much of the workload consists of short messages; the 11-byte and 20-byte figures are UEC specification claims, not a guaranteed application-level speedup.

Unified Local for scale-up endpoints

The new Unified Local profile targets scale-up endpoints and includes a simplified transport protocol supporting write, read, and atomic operations. Those verbs map directly to common accelerator communication needs: moving a tensor into a peer’s memory, fetching data, and updating synchronization or reduction state without forcing every operation through a heavier general-purpose sequence.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Use of the profile still requires matching support in the accelerator or NIC, switch pipeline, firmware, driver, and communication libraries. A switch that merely advertises high-speed Ethernet is not evidence that Unified Local is implemented end to end.

Collective-operation offload

Collectives often dominate training time because every participant must contribute to and receive a result. UEC’s example compares a traditional 1 GB all-reduce, in which each endpoint sends 2 GB, with in-network collective processing that reduces endpoint transmission to 1 GB. In that example, the reduced endpoint traffic can provide up to a 2x communication speedup. The figure is tied to the stated 1 GB all-reduce example; it is not a universal benchmark for every topology, tensor size, algorithm, or software stack.

Collective offload moves part of the reduction or distribution work into the network. Before selecting it, verify which collective algorithms, data types, reduction operators, and failure behaviors are supported by the implementation and framework. Unsupported operations may fall back to ordinary endpoint communication, producing mixed performance within one job.

Congestion Signaling and Programmable Congestion Management

Congestion Signaling (CSIG) reports information such as queueing latency, network delay, or queue depth. That telemetry gives an endpoint or control system a more useful view than packet loss alone, particularly when an AI job is sensitive to tail latency.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Rank #2
Sale
TP-Link TL-SX105, 5 Port 10G/Multi-Gig Unmanaged Ethernet Switch
  • 𝐅𝐢𝐯𝐞 𝟏𝟎𝐆𝐛𝐩𝐬 𝐏𝐨𝐫𝐭𝐬 𝐟𝐨𝐫 𝐋𝐢𝐠𝐡𝐭𝐧𝐢𝐧𝐠-𝐅𝐚𝐬𝐭 𝐂𝐨𝐧𝐧𝐞𝐜𝐭𝐢𝐨𝐧𝐬: 5× 10-Gigabit ports unlock the highest performance with 10G/multi-gig bandwidth and provide up to 100 Gbps of switching capacity.
  • 𝐀𝐮𝐭𝐨-𝐍𝐞𝐠𝐨𝐭𝐢𝐚𝐭𝐢𝐨𝐧: Auto-negotiation intelligently senses the link speeds and adjusts between 5-speeds (100Mb/1G/2.5G/5G/10G) for compatibility and optimal performance for all your devices, including 2.5G/5G/10G WiFi 6 AP, 10G NAS, 10G PCIe Adapter/NIC, 10G Server, gaming computer, 8K video, and more.
  • 𝐑𝐞𝐥𝐢𝐚𝐛𝐥𝐞 𝐚𝐧𝐝 𝐐𝐮𝐢𝐞𝐭: IEEE 802.3X flow control provides reliable data transfer and a fanless design ensures quiet operation.
  • 𝐏𝐥𝐮𝐠 𝐚𝐧𝐝 𝐏𝐥𝐚𝐲: Easy setup with no software installation or configuration needed.
  • 𝐒𝐭𝐮𝐫𝐝𝐲 𝐌𝐞𝐭𝐚𝐥 𝐂𝐚𝐬𝐞: Durable metal casing and desktop/wall-mounting design are well-suited for different environments.

Programmable Congestion Management (PCM) exposes APIs for programmable congestion and load-balancing algorithms. This allows an operator or vendor to tune behavior for a topology’s traffic mix instead of relying on one fixed policy. It also creates an operational requirement: algorithms, thresholds, telemetry collection, and rollback procedures must be tested under incast and burst conditions. Programmability is a capability, not an automatic guarantee of fairness or stability.

When to choose a scale-up-first design

Choose scale-up first when the dominant question is how quickly accelerators inside one pod can exchange data for the next synchronized operation.

  • The workload repeatedly exchanges tensors among accelerators in one pod or domain.
  • Microsecond-level unloaded latency and, especially, tail latency affect step time.
  • Messages include frequent reads, writes, or atomic synchronization operations.
  • All-reduce, all-gather, or related collectives consume a large share of communication time.
  • The platform can provide end-to-end support for Unified Local, UFH, CSIG, PCM, and any collective offload you plan to use.

UE 1.1 is a strong candidate for this layer because Unified Local addresses the local transport verbs, UFH and inter-packet-gap reductions reduce per-packet cost, and collective offload can reduce endpoint traffic. Do not select it solely because the product label says Ethernet 1.1; confirm the profiles and software paths that your model actually invokes.

When to choose a scale-out-first design

Choose scale-out first when the system must connect many independent endpoints and remain usable while jobs, tenants, and paths change.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  • Training spans multiple pods or racks and requires routed reachability between them.
  • The endpoint count is large enough that multipathing, load balancing, and failure isolation dominate design decisions.
  • Traffic is bursty or incast-heavy, with many workers sending toward a smaller set of receivers.
  • Several jobs or tenants share the same fabric and require predictable service boundaries.
  • Operations teams need standard Ethernet routing, telemetry, and staged fault recovery.

Ultra Ethernet’s earlier specification work focused on backend scale-out. UEC deployment models list fabrics from tens of thousands to 256,000 Ethernet ports, illustrating the scale the architecture is intended to address. At this level, a small latency improvement inside one pod cannot compensate for poor congestion behavior or a fabric that cannot drain a failed path.

Published UEC deployment characteristics

The following figures are planning characteristics published by the Ultra Ethernet Consortium. They are not guarantees for every vendor implementation, topology, workload, or software release.

Deployment model Port scale Port speed Listed unloaded one-way latency target Utilization characteristic
Dedicated AI training cluster 100,000–256,000 Ethernet ports 800G+ 2–10 microseconds Up to 85% average network utilization
Cloud AI/HPC Up to 256,000 Ethernet ports 400G+ 2–10 microseconds 20–40% overall; 60–80% for AI cloud
At-scale HPC 80,000–256,000 Ethernet ports 800G+ 2–10 microseconds Not separately stated

“Unloaded” latency excludes the queueing and contention that often determine application tail latency. Treat these numbers as architecture targets for the listed deployment models, not as a promise that a completed cluster will deliver them under load.

Rank #3
TP-Link TL-SG105S-M2, 5 Port Multi-Gigabit 2.5G Unmanaged Ethernet Switch | Ethernet Splitter, Plug & Play, Desktop/Wall Mount, Silent Operation
  • 𝗙𝗶𝘃𝗲 𝟮.𝟱 𝗚𝗯𝗽𝘀 𝗣𝗼𝗿𝘁𝘀 𝗳𝗼𝗿 𝗦𝘂𝗽𝗲𝗿-𝗙𝗮𝘀𝘁 𝗖𝗼𝗻𝗻𝗲𝗰𝘁𝗶𝗼𝗻𝘀: 5× 2.5-Gigabit ports unlock the highest performance of your Multi-Gig bandwidth and devices, and provide up to 25 Gbps of switching capacity.
  • 𝗔𝘂𝘁𝗼-𝗡𝗲𝗴𝗼𝘁𝗶𝗮𝘁𝗶𝗼𝗻: Auto-negotiation intelligently senses the link speeds and adjusts between 3-speeds (100Mb/1G/2.5G) for compatibility and optimal performance for all your devices, including 2.5G WiFi 6 AP, 2.5G NAS, 2.5G PCIe Adapter, 2.5G Server, gaming computer, 4K video, and more.
  • 𝗜𝗱𝗲𝗮𝗹 𝗳𝗼𝗿 𝗩𝗮𝗿𝗶𝗼𝘂𝘀 𝗦𝗰𝗲𝗻𝗮𝗿𝗶𝗼𝘀: Built for LAN parties, home entertainment, small and home offices, and instant transfer for workstations.
  • 𝗛𝗮𝘀𝘀𝗹𝗲-𝗙𝗿𝗲𝗲 𝗖𝗮𝗯𝗹𝗶𝗻𝗴: Instantly upgrade to 2.5 Gbps without the need to upgrade to Cat6 wiring, reducing wiring costs and hassle. *
  • 𝗦𝗶𝗹𝗲𝗻𝘁 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻: Industry-leading fanless design ensures silent operation, ideal for any home or business.

Why most large clusters use a hybrid

A hybrid design keeps the shortest, most synchronized conversations on a scale-up fabric and uses scale-out Ethernet to connect pods, racks, storage, and services. A training job can therefore use low-latency local operations inside each pod while still growing beyond one pod.

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Typical traffic placement

  • Inside a pod: place accelerator-to-accelerator reads, writes, atomics, and latency-sensitive collectives on the scale-up path.
  • Between pods: use the scale-out fabric for routed traffic, multipathing, congestion isolation, and broad endpoint reach.
  • At the boundary: verify that the NICs and libraries preserve ordering, semantics, and congestion information when traffic crosses layers.
  • For storage and control traffic: keep independent queues or virtual networks where necessary so checkpointing, monitoring, and orchestration cannot starve training traffic.

The boundary is an engineering choice, not a requirement that every packet use one protocol. Some platforms may expose one programming model over both layers; others may select different transports based on message size and destination.

Ultra Ethernet 1.1 versus UALink

UALink is a specialized scale-up accelerator interconnect. Its official 1.0 specification page states 200G per lane and support for up to 1,024 accelerators within an AI computing pod. UE 1.1 is broader: it defines an Ethernet-based communications stack that includes transport, congestion behavior, collective operations, and interoperability for AI and HPC fabrics.

Question UE 1.1 UALink 1.0
Primary role Scale-up and scale-out Ethernet networking, with a new local profile for scale-up Dedicated scale-up accelerator interconnect
Published bandwidth detail Deployment models list 400G+ or 800G+ Ethernet ports; per-lane figure not stated here 200G per lane
Published pod scale Deployment models list up to 256,000 Ethernet ports across large fabrics Up to 1,024 accelerators in an AI computing pod
Routing scope Retains IP routing and Ethernet interoperability through the UE stack Specialized local accelerator domain
Congestion and collectives CSIG, PCM, and in-network collective features are specified Capabilities depend on the relevant UALink implementation; no equivalent figures are established here

UE 1.1 is not automatically a replacement for UALink. A platform may use UALink or another dedicated scale-up interconnect inside a pod and UE-based Ethernet between pods. Conversely, a vendor may implement UE 1.1 for both local and remote communication. The deciding factors are available silicon, firmware, drivers, libraries, topology, and the operator’s need for Ethernet interoperability.

Ultra Ethernet 1.1 versus InfiniBand

The Ultra Ethernet Consortium’s published materials do not establish a like-for-like latency, bandwidth, utilization, or cost comparison between InfiniBand and UE 1.1, so claims that one is universally faster or cheaper would be unsupported.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

The practical comparison is architectural. UE 1.1 uses Ethernet framing, routing, and interoperability as part of its design and adds AI-oriented transport, congestion, and collective functions. An InfiniBand-based design may offer a mature, purpose-built fabric with its own adapters, switches, management, and software ecosystem. Compare the complete implementation rather than the specification name: collective-library support, failure recovery, telemetry, topology limits, power, cabling, operational skills, and migration effort all affect the result.

A step-by-step selection process

  1. Map communication locality. Record which messages stay inside one accelerator pod, cross pods in one rack, cross racks, or leave the cluster. Do not size the whole fabric from an average; identify the busiest and most latency-sensitive paths.
  2. Characterize the workload. Measure message sizes, packet rate, all-reduce and all-to-all frequency, synchronization behavior, incast, and tail latency. A workload dominated by tiny atomics has different needs from one dominated by large streaming transfers.
  3. Choose the layer for each flow. Assign local accelerator traffic to a scale-up path when microsecond-level behavior and collective efficiency matter. Assign inter-pod and shared-service traffic to scale-out Ethernet when routing, multipathing, and isolation matter.
  4. Verify implementation support. Check the exact NIC, switch ASIC, optics, firmware, driver, kernel or user-space stack, and communication library. Confirm Unified Local, UFH, CSIG, PCM, and collective offload support individually; “UE 1.1 compatible” alone is not a product certification.
  5. Validate congestion behavior. Reproduce synchronized bursts, incast, link failures, and competing jobs. Inspect queue depth, queueing latency, network delay, retransmission or recovery behavior, and tail latency rather than only average throughput.
  6. Test interoperability. Confirm that the intended UE endpoints, routed Ethernet sections, gateways, and management systems exchange traffic correctly. Include mixed generations and fallback paths if the deployment will contain them.
  7. Calculate total operating cost. Include switches, NICs, optics, cabling, power, cooling, software licensing where applicable, monitoring, staff expertise, migration work, and the cost of reserving capacity for failures.
  8. Run application acceptance tests. Measure completed training-step time, collective time, checkpoint impact, job-to-job interference, and recovery after a failed link or switch. A network result matters only if the model or HPC application benefits.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Implementation pitfalls and recovery actions

Assuming a specification feature is enabled everywhere

A specification defines behavior; it does not certify a particular product combination. If UFH or Unified Local traffic falls back to a generic path, packet overhead and latency may differ from the design estimate. Build a capability matrix for every hop and expose negotiated or configured modes in monitoring.

Rank #4
Sale
TP-Link TL-SG105, 5 Port Gigabit Unmanaged Ethernet Switch | Network Hub, Ethernet Splitter, Plug & Play, Fanless Metal Design, Shielded Ports, Traffic Optimization
  • 𝗢𝗻𝗲 𝗦𝘄𝗶𝘁𝗰𝗵 𝗠𝗮𝗱𝗲 𝘁𝗼 𝗘𝘅𝗽𝗮𝗻𝗱 𝗡𝗲𝘁𝘄𝗼𝗿𝗸: 5× 10/100/1000Mbps RJ45 Ports supporting Auto Negotiation and Auto MDI/MDIX.
  • 𝗚𝗶𝗴𝗮𝗯𝗶𝘁 𝘁𝗵𝗮𝘁 𝗦𝗮𝘃𝗲𝘀 𝗘𝗻𝗲𝗿𝗴𝘆: Latest innovative energy-efficient technology greatly expands your network capacity with much less power consumption and helps save money.
  • 𝗥𝗲𝗹𝗶𝗮𝗯𝗹𝗲 𝗮𝗻𝗱 𝗤𝘂𝗶𝗲𝘁: IEEE 802.3X flow control provides reliable data transfer and Fanless design ensures quiet operation.
  • 𝗣𝗹𝘂𝗴 𝗮𝗻𝗱 𝗣𝗹𝗮𝘆: Easy setup with no software installation or configuration needed.
  • 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗙𝗲𝗮𝘁𝘂𝗿𝗲𝘀: Prioritize your traffic and guarantee high quality of video or voice data transmission with Port-based 802.1p/DSCP QoS and IGMP Snooping.

Planning from unloaded latency alone

The published 2–10 microsecond figure is an unloaded one-way target. Under synchronized training bursts, queues and incast can dominate. If tail latency spikes, inspect queue depth and CSIG telemetry, then adjust PCM policies, traffic classes, load balancing, or topology instead of chasing a lower idle-latency number.

Expecting collective offload to accelerate every reduction

The UEC example shows up to 2x communication speedup for a specific 1 GB all-reduce traffic comparison. Unsupported operators, data types, message sizes, or topologies can use endpoint fallback. Confirm which collectives are offloaded and compare end-to-end step time.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Using scale-out controls to solve a scale-up locality problem

More routing capacity between racks does not remove per-packet overhead or local synchronization delays inside a pod. If profiling shows local traffic dominates, improve the scale-up path and its library mapping before adding scale-out links.

Using a local fabric as a universal cluster network

A specialized pod interconnect may not provide the routing, tenant isolation, or fault containment needed across many racks. Keep a scale-out design for broad reach, and test what happens when a pod, uplink, or gateway is unavailable.

Ignoring software and operational dependencies

Drivers, firmware, collective libraries, telemetry pipelines, and congestion algorithms determine whether the hardware capabilities are usable. Stage upgrades, preserve a known-good fallback, and document rollback steps before changing a production fabric.

Minimum acceptance checklist

  • Communication map identifies inside-pod, inter-pod, inter-rack, storage, and control flows.
  • Workload profile includes message size, packet rate, collective mix, incast, average latency, and tail latency.
  • Every NIC, switch, optical module, firmware image, driver, and library has a verified UE 1.1 feature matrix.
  • Unified Local, UFH, CSIG, PCM, and collective offload are tested separately rather than inferred from a single compatibility label.
  • Congestion tests include synchronized bursts, competing jobs, link failures, and recovery.
  • Interoperability with routed Ethernet and any dedicated scale-up interconnect is demonstrated on the intended topology.
  • Total cost includes power, cooling, cabling, operations, migration, and spare capacity.
  • Application tests report training-step or solver completion time, not only synthetic bandwidth.

Bottom line for architecture decisions

Use UE 1.1’s scale-up features when accelerators in one pod need efficient reads, writes, atomics, and collectives with minimal packet overhead. Use its scale-out capabilities when the priority is a routed, congestion-aware Ethernet fabric spanning many pods and racks. For most large AI systems, the defensible design is hybrid: keep tightly synchronized local traffic on the scale-up layer, carry broad cluster traffic on scale-out Ethernet, and select UALink, UE 1.1, InfiniBand, or a combination only after verifying end-to-end hardware and software support.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

AI networkingEthernetHPCInfiniBandscale-out networkingscale-up networkingUALinkUltra Ethernet 1.1

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Recommended PC Tool
Recommended PC Tool
PC Slower Than It Used to Be?Free scan - under a minute
Crashes, No Sound, or Screen Glitches?Free driver scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.