TL;DR

An NVIDIA Spectrum-X fabric should not be approached as a conventional Ethernet refresh with faster switches. Distributed AI creates synchronized, high-bandwidth traffic patterns in which congestion, packet loss, path imbalance, and tail latency can slow an entire training job.

A production design should:

• Separate the high-performance compute fabric from storage, in-band services, and out-of-band management.
• Use a routed leaf-spine topology with enough uplink capacity to meet the intended blocking ratio.
• Align GPU-facing network interfaces into predictable rails.
• Treat RoCE as an end-to-end system covering hosts, SuperNICs, switches, queue mappings, ECN, PFC, congestion control, and MTU.
• Use BGP and ECMP as the underlay, then apply adaptive routing where the validated Spectrum-X design supports it.
• Validate cabling, optics, FEC, lane mappings, and traffic balance before accepting the fabric.
• Monitor microbursts, ECN marking, PFC activity, buffer occupancy, link errors, and route health through NetQ and high-frequency telemetry.
• Design maintenance and failure domains before the first production AI job is scheduled.

The most important architectural principle is simple: build the network as a coordinated AI transport system, not as a collection of independent Ethernet devices.

Introduction

A distributed AI workload can place very different demands on Ethernet than a conventional enterprise application.

A typical application environment contains a mixture of short transactions, north-south requests, storage access, backup traffic, user sessions, and background services. Individual flows may be important, but the application is often designed to tolerate variation between them.

Distributed GPU training behaves differently. Hundreds or thousands of accelerators may participate in collective operations in which each worker must exchange data before the next stage can continue. A single congested path, misconfigured queue, flapping optic, or slow rail can delay the wider job.

That changes the network design question.

The objective is not simply to provide a large amount of aggregate bandwidth. The fabric must deliver predictable, balanced, low-loss communication across every active path while remaining observable and maintainable at scale.

NVIDIA Spectrum-X combines Spectrum Ethernet switches, NVIDIA networking adapters or SuperNICs, Cumulus Linux, RoCE transport, congestion controls, adaptive routing, and telemetry into an integrated architecture for this problem. However, deploying the individual components does not automatically create a well-designed AI fabric.

The design still depends on topology, separation, capacity, configuration consistency, cabling discipline, operational ownership, and acceptance testing.

This article develops that implementation path.

Design Assumptions and Scope

The examples in this article use the following assumptions:

• The environment is an on-premises or colocation AI factory using Ethernet for GPU scale-out communication.
• Spectrum-4 switches and BlueField-3 SuperNICs are used as the representative hardware generation.
• The compute fabric uses routed leaf-spine connectivity.
• BGP provides the Layer 3 underlay.
• RoCEv2 carries RDMA traffic between compute nodes.
• Out-of-band management remains physically independent from the high-performance fabric.
• Storage may use a separate high-performance fabric or a logically isolated converged network, depending on scale and performance requirements.
• Configuration examples use current NVUE syntax as an implementation illustration.
• Production versions must be pinned to an NVIDIA-validated solution stack, server platform, adapter firmware, Cumulus Linux release, and reference architecture.

Newer NVIDIA systems may use different adapters, switch speeds, port modes, or validated software versions. The architecture remains useful, but the bill of materials and exact configuration must follow the reference architecture for the selected GPU platform.

Why Conventional Data-Center Ethernet Can Struggle With Distributed AI

Conventional Ethernet is not inherently unsuitable for AI. The problem is that a general-purpose data-center network is usually optimized for a broader and less synchronized traffic mix.

Distributed AI exposes weaknesses that may remain invisible under ordinary application traffic.

Persistent Elephant Flows

AI jobs can create large, long-running flows between many GPU nodes. These flows remain active long enough to collide repeatedly on the same ECMP path.

A flow-based hash can distribute connections across several links, but it does not continuously evaluate whether those links remain equally loaded. Two or more large flows can hash onto one path while another path remains underused.

Low-Entropy Traffic Patterns

Some collective operations generate relatively few large flows between predictable endpoint pairs. This gives an ECMP hash fewer independent flow combinations to distribute.

The network may contain several equal-cost paths while still failing to use them evenly.

Incast and Many-to-One Congestion

During synchronized communication, multiple nodes may send toward the same destination or group of destinations at nearly the same time.

The receiving port can become congested even when the wider fabric has available capacity. Adaptive path selection helps before the destination, but it cannot remove congestion at the final receiver. End-to-end congestion control is still required.

Packet Loss Has an Amplified Cost

A dropped packet does not affect only the packet itself. It can trigger retransmission, delay completion of a collective operation, and leave expensive GPUs waiting.

The resulting business impact is not accurately represented by ordinary network utilization alone. A fabric can look lightly utilized on average while short bursts repeatedly delay training iterations.

Tail Latency Matters More Than Average Latency

In synchronized workloads, the slowest participating communication path can determine when the collective operation completes.

Average latency can therefore look healthy while a small number of delayed flows reduce effective job performance.

Design concern	General-purpose data-center fabric	AI-optimized Spectrum-X fabric
Dominant traffic	Mixed short and long flows	Large synchronized east-west flows
Path selection	Primarily static per-flow ECMP	ECMP plus adaptive routing where supported
Congestion response	Often reactive after queue growth or loss	ECN, source congestion control, queue telemetry, and selective PFC
Loss tolerance	Usually acceptable within application retry limits	Loss can delay an entire distributed operation
Capacity planning	Aggregate utilization and oversubscription	Per-rail capacity, incast, blocking ratio, and collective behavior
Telemetry	Periodic counters and general alerts	High-frequency queue, ECN, PFC, path, and fabric telemetry
Operational objective	Maintain application reachability	Maintain balanced and predictable GPU communication

The practical lesson is that link speed alone does not solve AI networking. A faster but poorly balanced fabric can still leave accelerators idle.

Separate the Compute, Storage, In-Band, and Management Fabrics

The first architecture decision should be traffic separation, not switch model selection.

An AI factory normally contains at least four different network functions. They have different availability requirements, traffic characteristics, security boundaries, and maintenance dependencies.

Compute Fabric

The compute fabric carries high-performance east-west GPU communication.

Its priorities are:

• Predictable low latency.
• High bisection bandwidth.
• RoCE support.
• Consistent queue configuration.
• Balanced path utilization.
• Minimal interference from unrelated traffic.
• Fast isolation of unhealthy links and switches.

This is the fabric most directly associated with Spectrum-X.

Storage Fabric

The storage fabric carries model data, checkpoints, training data, and results.

High-performance storage can also use RoCE and may require similar loss and congestion characteristics. However, storage traffic should not be assumed to behave like GPU collective traffic.

A separate storage fabric provides clearer capacity and failure boundaries. A converged design can reduce hardware count, but it requires disciplined segmentation, bandwidth guarantees, queue planning, and operational coordination.

In-Band Network

The in-band network supports services used by compute nodes and cluster software, such as:

• Operating system access.
• Cluster management services.
• Package and container repositories.
• Authentication and directory services.
• Monitoring agents.
• Lower-performance file services.
• User access and data-ingestion services.
• Connections to external data-center networks.

This traffic usually does not require the same treatment as the GPU compute fabric.

Out-of-Band Management Network

The out-of-band network connects management interfaces that must remain available when the production fabric is impaired.

Typical endpoints include:

• Server baseboard management controllers.
• SuperNIC or DPU management interfaces.
• Switch management ports.
• Console servers.
• Power distribution units.
• Rack-management appliances.

The out-of-band network should be physically independent. Sharing the same failure domain with the network being repaired defeats its purpose.

Fabric	Primary purpose	Typical transport	Recommended separation
Compute	GPU-to-GPU scale-out traffic	RoCEv2	Dedicated physical fabric
High-performance storage	Training data and checkpoint movement	Ethernet, RoCE, or storage-specific transport	Dedicated at larger scale or when performance requires it
In-band	Host, cluster, repository, monitoring, and user services	Standard IP Ethernet	Separate physical or strongly segmented converged fabric
Out-of-band	Hardware management and recovery	Standard management Ethernet	Physically independent

Logical VLAN or VRF separation can be appropriate for some in-band and storage services. It should not be treated as equivalent to physical independence when capacity contention, maintenance isolation, or recovery access is important.

Use a Nonblocking Leaf-Spine Fabric

A leaf-spine topology gives every leaf a consistent number of routed hops to every other leaf.

Each compute-facing leaf connects to every spine in its plane. Traffic between two compute nodes attached to different leaves follows this general path:

The architecture is simple, but the capacity calculation is often misunderstood.

Calculate the Blocking Ratio Explicitly

For a fully nonblocking leaf:

Total leaf uplink bandwidth >= Total active host-facing bandwidth

A leaf with 16 active 400 Gb/s host-facing links has:

16 x 400 Gb/s = 6.4 Tb/s host-facing bandwidth

To remain nonblocking at the leaf, it requires at least 6.4 Tb/s of usable spine-facing bandwidth.

A design with 4.8 Tb/s of uplink bandwidth has:

Blocking ratio = 6.4 / 4.8
               = 1.33:1

That does not automatically make the design invalid. It means the oversubscription is an explicit design decision that must be tested against the expected collective communication pattern.

The calculation should account for:

• Active ports, not only installed ports.
• Per-port operating speed.
• Breakout mode.
• Planned link failures.
• Maintenance conditions.
• Growth headroom.
• Whether one plane must carry traffic after another plane fails.
• Storage or service traffic sharing the same hardware.

A fabric described as nonblocking should remain nonblocking under its documented normal operating condition. It should also state what performance is expected during a failed-link or failed-switch condition.

Organize the Fabric Around Network Rails

Rail optimization groups corresponding GPU-facing interfaces from multiple servers onto aligned leaf-switch groups.

The objective is to keep common collective communication paths predictable and reduce unnecessary traversal between unrelated parts of the topology.

The following simplified design uses four rails. It is an architectural example, not a fixed server-port prescription.

The important design discipline is consistency.

NIC 0 on one compute node should not accidentally land on the rail intended for NIC 3 on another. Cabling records, switch-port descriptions, rack elevations, adapter enumeration, and operating system interface names must agree.

A rail-optimized design does not remove the need for the spine layer. Traffic within an aligned rail may remain localized under some communication patterns, while traffic between rail groups traverses the spine.

Decide Whether the Fabric Needs One Plane or Two

A single-plane design can be fully functional and may be appropriate for smaller scale units.

A dual-plane design provides stronger isolation. Each compute node connects to both planes through independent adapters, switch groups, cabling paths, and preferably separate power domains.

Potential benefits include:

• Continued connectivity after a plane failure.
• Maintenance without removing every network path.
• Reduced blast radius for switch software or configuration problems.
• Independent path testing.

However, dual-plane should not be described as automatic full-performance redundancy.

If one plane fails, the job may continue with less available bandwidth. The actual outcome depends on:

• Adapter and collective-library behavior.
• Remaining path capacity.
• Job topology.
• Whether traffic is redistributed quickly.
• Whether the surviving plane was sized for degraded operation.
• Whether the failure occurred during an active collective operation.

The design document should therefore state two separate objectives:

1. Connectivity objective: Can the job continue?
2. Performance objective: What throughput remains after the failure?

Match Spectrum Switches and BlueField SuperNICs as a System

Spectrum-X is an end-to-end architecture. The value does not come from treating the switch and adapter as unrelated purchases.

Spectrum-4 Switch Role

Spectrum-4 switches provide the high-bandwidth leaf and spine layer.

Depending on the selected platform and port mode, the hardware may support combinations of 800 Gb/s, 400 Gb/s, and breakout connectivity. The bill of materials must identify:

• Switch model.
• Port operating mode.
• Host-facing speed.
• Spine-facing speed.
• Breakout requirements.
• Connector type.
• Cable and optic compatibility.
• Required airflow.
• Power supply redundancy.
• Rack power and cooling.
• Cumulus Linux and firmware versions.

A port-count calculation that ignores breakout lanes and connector constraints is not sufficient.

BlueField-3 SuperNIC Role

BlueField-3 SuperNICs provide high-speed Ethernet, RDMA, and RoCE capabilities at the compute node.

They are particularly relevant to east-west AI communication and can participate in the end-to-end congestion-control architecture. They also support technologies that reduce unnecessary data movement through the host CPU, including GPU-oriented data paths.

The exact mode matters. A BlueField device used as a SuperNIC for scale-out networking should not be assumed to have the same operational role as a DPU deployed for infrastructure offload, security services, or north-south processing.

Document the intended role before deployment:

Design field	Example decision
Device role	AI compute SuperNIC
Port speed	400 Gb/s
Number of paths	One per rail or plane as defined by the server architecture
RDMA transport	RoCEv2
Host driver baseline	Pinned to validated solution
Adapter firmware	Pinned to validated solution
Congestion-control profile	NVIDIA-validated profile
Management ownership	AI platform and network teams jointly defined

Newer reference architectures may use newer adapter generations. Do not combine hardware, firmware, and software versions from different validated designs because the individual components appear compatible on paper.

Build a Routed BGP Underlay

A routed Layer 3 fabric provides predictable failure isolation and avoids extending large Layer 2 domains through the spine.

A common pattern uses eBGP between each leaf and spine interface.

Underlay Design Principles

Use the following principles:

• Every leaf connects to every spine in its plane.
• Leaf-spine links are routed interfaces.
• Each switch has a stable router ID.
• Autonomous system numbering follows a documented convention.
• ECMP installs multiple equal-cost paths.
• Route policies are intentionally limited.
• Infrastructure addressing can be generated from a repeatable plan.
• BGP sessions, route counts, and path counts are monitored.
• No individual spine becomes a mandatory transit dependency.

Interface-based BGP unnumbered can reduce point-to-point address consumption and simplify configuration, but it should still be generated and validated consistently.

Illustrative NVUE BGP Configuration

The following is an underlay skeleton for a leaf with two spine-facing ports. It is not a complete production configuration.

# Assign the leaf autonomous system and router ID.
nv set router bgp autonomous-system 65101
nv set router bgp router-id 10.255.1.1

# Form eBGP relationships over the spine-facing interfaces.
nv set router bgp neighbor swp51 remote-as external
nv set router bgp neighbor swp52 remote-as external

# Commit the candidate configuration.
nv config apply

# Review the resulting BGP configuration and state.
nv show router bgp

The values that must change include:

• Leaf autonomous system.
• Router ID.
• Spine-facing interface names.
• Route origination or redistribution policy.
• Address-family behavior.
• Peer groups.
• Authentication, if required.
• BFD and convergence settings, when part of the validated design.

Successful execution means every intended leaf-spine adjacency is established, the expected routes are installed, and every endpoint has the required number of equal-cost paths.

A BGP session being established does not prove that the fabric is correct. The validation must also confirm next-hop resolution, route symmetry, ECMP width, failure convergence, and endpoint reachability.

Treat RoCE as an End-to-End Profile

RoCE configuration is frequently reduced to a switch command. That is an incomplete implementation model.

A working RoCE fabric requires alignment across:

• Server operating system.
• Adapter driver.
• Adapter firmware.
• BlueField configuration.
• Traffic marking.
• Switch trust behavior.
• Queue mapping.
• ECN thresholds.
• PFC priorities.
• Congestion-control behavior.
• MTU.
• Routing.
• Telemetry.
• Application or collective-communication libraries.

A mismatch at any point can create a fabric that passes basic connectivity tests but performs poorly under synchronized load.

Cumulus Linux RoCE Baseline

Current Cumulus Linux provides an NVUE RoCE profile that configures the switch-side quality-of-service baseline.

# Enable the NVIDIA RoCE QoS profile.
nv set qos roce

# Apply the configuration.
nv config apply

# Review the resulting QoS state.
nv show qos

The default lossless profile maps RoCE traffic and congestion-notification traffic into distinct traffic classes. It enables PFC only for the RoCE priority rather than making every class lossless.

That narrow scope is important.

PFC should be treated as a controlled safety mechanism for the selected lossless class. Enabling pause behavior across unrelated traffic classes can allow congestion in one workload to block management, storage, or service traffic.

Do not copy custom queue thresholds from another switch model or software version. Buffer allocation and threshold behavior can be ASIC-specific.

Understand the Role of ECN, PFC, Congestion Control, and Adaptive Routing

These mechanisms solve different parts of the congestion problem. They are complementary, not interchangeable.

ECN Provides an Early Congestion Signal

Explicit Congestion Notification allows a switch to mark packets when queue occupancy crosses a configured threshold.

The receiver communicates that signal back to the sender through congestion-notification traffic. The sending adapter can then reduce its transmission rate before the congested queue begins dropping packets.

ECN is an early-warning mechanism. It does not directly select a different path.

PFC Protects a Selected Priority

Priority Flow Control can pause a specific priority on a specific link.

It is useful as a short-term loss-avoidance mechanism, but it introduces operational risks when overused. Poorly scoped PFC can propagate backpressure, increase head-of-line blocking, or make a congested receiver affect a larger part of the fabric.

Use PFC only for the intended RoCE priority, and monitor pause duration and frequency.

A healthy fabric should not depend on sustained PFC activity as its normal congestion-control strategy.

Congestion Control Adjusts the Sender

Many-to-one congestion can occur at the destination-facing link. Adaptive routing cannot eliminate that final bottleneck because every valid path eventually reaches the same receiver.

The sender must react to congestion feedback and adjust its injection rate.

That makes adapter firmware, host configuration, and switch ECN behavior part of the same control loop.

Adaptive Routing Selects Less-Loaded Paths

ECMP selects from equal-cost paths using a hash. Adaptive routing can use current path conditions, including link utilization or queue state, to choose a less-loaded alternative.

The relationship can be visualized as follows:

Illustrative Adaptive Routing Configuration

Adaptive routing should be enabled only where supported by the switch generation, port speed, topology, and validated reference design.

# Enable adaptive routing globally.
nv set router adaptive-routing state enabled

# Enable it on selected routed fabric interfaces.
nv set interface swp51 router adaptive-routing state enabled
nv set interface swp52 router adaptive-routing state enabled

# Apply the configuration.
nv config apply

# Review adaptive-routing state.
nv show router adaptive-routing

Enabling or disabling adaptive routing can restart or reload switching processes, depending on the software behavior. Plan it as a controlled change rather than a harmless live toggle.

Current support also has interface and speed constraints. For example, a feature supported on routed 200 Gb/s or 400 Gb/s links should not be assumed to work identically on 800 Gb/s ports, bonds, switch virtual interfaces, or subinterfaces.

Pin the implementation to the supported matrix for the deployed Cumulus Linux version.

Make the MTU Decision End to End

Cumulus Linux commonly uses a large default switch-port MTU, but the important requirement is not the number on one switch.

The requirement is consistency across:

• Compute-node interface.
• BlueField or NIC port.
• Leaf host-facing port.
• Leaf-spine link.
• Spine port.
• Destination leaf.
• Destination host.
• Any routed boundary traversed by the traffic.

An MTU mismatch can cause packet loss, fragmentation, or confusing performance degradation that appears only with larger payloads.

Illustrative NVUE MTU Configuration

# Configure a host-facing port.
nv set interface swp1 link mtu 9216

# Configure a spine-facing port.
nv set interface swp51 link mtu 9216

# Apply and inspect the configuration.
nv config apply
nv show interface

The server-side MTU may use a different displayed value depending on operating-system and adapter conventions. Validate the effective end-to-end payload rather than assuming that matching-looking numbers prove compatibility.

Acceptance testing should include:

• Large nonfragmented IP packets.
• RoCE traffic at expected message sizes.
• Traffic across every rail.
• Traffic across every spine path.
• Testing during a failed-link condition.
• Verification that counters remain free from MTU-related drops.

Design the Lossless Scope Deliberately

Lossless Ethernet should not mean that the entire fabric pauses whenever any queue is congested.

A better design is selective:

• RoCE is mapped to a defined priority.
• PFC is enabled only for that priority.
• Congestion-notification traffic receives appropriate scheduling treatment.
• Ordinary traffic remains in lossy queues unless another requirement justifies different handling.
• Storage traffic receives its own explicit classification.
• Management traffic is protected from RoCE backpressure.
• Queue and headroom usage is measured during realistic workloads.

The network team should document the mapping from application marking to physical queue.

Layer	Decision to document
Application or host	DSCP or priority marking
SuperNIC	Trust, rewrite, and congestion-control behavior
Leaf ingress	Classification and traffic-class mapping
Switch queue	Scheduling, ECN, buffer, and PFC behavior
Leaf-spine links	Preservation of markings
Destination	Congestion notification and receiver behavior
Monitoring	ECN, CNP, PFC, occupancy, and drop counters

Without this mapping, troubleshooting becomes a debate between teams rather than a traceable packet path.

Plan Cabling and Optics as Part of the Architecture

At 400 Gb/s and 800 Gb/s, cabling is not a finishing task. It is part of the system design.

The bill of materials should identify:

• Switch port type.
• Adapter port type.
• Cable or transceiver model.
• Supported reach.
• Fiber type.
• Connector type.
• Breakout mapping.
• Lane count.
• Polarity.
• FEC requirements.
• Auto-negotiation requirements.
• Power consumption.
• Thermal constraints.
• Minimum bend radius.
• Approved combinations from the hardware compatibility documentation.

Spectrum-4 systems may expose high-density ports that break out into multiple lower-speed links. A physically connected cable does not prove that the lane mapping matches the intended logical ports.

Use a Deterministic Labeling Standard

A useful cable label should identify more than two device names.

For example:

PlaneA-Rail2-Node014-NIC2-Leaf07-swp18

The label allows an operator to infer:

• Fabric plane.
• Network rail.
• Compute node.
• Adapter position.
• Leaf switch.
• Switch port.

The same identifiers should appear in:

• Rack elevations.
• Cable schedules.
• Switch interface descriptions.
• Server inventory.
• Automation source data.
• Monitoring dashboards.
• Incident tickets.

Validate Layer 1 Before BGP

Do not begin routing troubleshooting until the physical layer has passed.

An initial switch-side check can include:

# Review interfaces, operating state, and counters.
nv show interface

# Review platform and hardware state.
nv show platform

# Review the intended port speed configuration.
nv set interface swp1 link speed 400G
nv config apply

Validate:

• Expected link speed.
• Auto-negotiation state.
• FEC mode.
• Transceiver recognition.
• Receive and transmit power where available.
• Symbol or lane errors.
• Link flaps.
• LLDP neighbor identity.
• Breakout port mapping.
• Cable polarity.
• Correct rail placement.

A link that transitions to the up state but accumulates FEC corrections or symbol errors should not be accepted into an AI fabric.

Use NVUE as the Configuration Interface

NVUE provides a structured configuration model across interfaces, routing, QoS, telemetry, and system functions.

This is useful for AI fabric automation because it reduces dependence on ad hoc file editing and makes configurations easier to generate consistently.

A practical workflow is:

Use the same source data to generate:

• Hostnames.
• Management addresses.
• Autonomous systems.
• Router IDs.
• Interface roles.
• Rail assignments.
• Port speeds.
• MTUs.
• BGP neighbors.
• RoCE profiles.
• Adaptive-routing scope.
• Interface descriptions.
• Monitoring metadata.

This is where the optional DevOps dimension becomes relevant. The article remains an architecture deep dive, but the production implementation should use version control, peer review, automated validation, and staged promotion.

Configuration Promotion Model

Use separate states for:

• Lab.
• Scale-unit pilot.
• Preproduction.
• Production.
• Maintenance candidate.

A configuration should not move directly from a template change to the complete production fabric.

Recommended gates include:

1. Syntax validation.
2. Topology validation.
3. Peer review.
4. Lab deployment.
5. Route and QoS validation.
6. Controlled pilot.
7. Performance testing.
8. Rollback test.
9. Production change approval.

Monitor the Fabric With NetQ and High-Frequency Telemetry

Traditional five-minute polling is too slow for many AI-fabric events.

A queue can fill, trigger pause behavior, recover, and appear normal before the next polling interval. Operators then see a slow job but no obvious network fault.

NetQ provides topology, route, interface, state-change, and troubleshooting visibility across the Cumulus Linux environment. It should be combined with high-frequency switch telemetry and workload-level observations.

Telemetry Signals to Collect

At minimum, collect:

• Interface state changes.
• Link flaps.
• Speed and FEC state.
• Symbol and physical errors.
• BGP adjacency changes.
• Route and ECMP path counts.
• Traffic distribution across equal-cost links.
• Queue occupancy.
• Queue watermarks.
• ECN-marked packets.
• Congestion-notification traffic.
• PFC pause frames.
• PFC pause duration.
• Buffer headroom usage.
• Adaptive-routing state.
• RoCE drops.
• Workload or collective-operation duration.
• GPU idle time attributable to communication.

Enable AI Ethernet and Buffer Telemetry

The following NVUE example enables telemetry exports for AI Ethernet and buffer statistics.

nv set system telemetry ai-ethernet-stats export state enabled
nv set system telemetry buffer-stats export state enabled
nv config apply

High-frequency telemetry can expose short-lived events that ordinary polling misses, including queue buildup, pause behavior, and uneven path utilization.

Use What Just Happened for Drop Analysis

Cumulus Linux also provides What Just Happened telemetry for supported drop categories.

An illustrative configuration can enable Layer 1 and buffer triggers on a named channel:

nv set system wjh channel fabric trigger l1
nv set system wjh channel fabric trigger buffer
nv config apply

This does not replace packet capture or application telemetry. It gives the operator another way to correlate switch-detected events with a job slowdown.

Build Operational Views Around Questions

Dashboards should answer operational questions rather than display every available counter.

Useful questions include:

• Is traffic evenly distributed across all spines?
• Is one rail consistently hotter than the others?
• Did ECN marking increase before job performance fell?
• Is PFC being used briefly or continuously?
• Are buffer watermarks approaching headroom limits?
• Did a BGP path disappear?
• Is one optic accumulating errors?
• Did a maintenance event reduce the expected ECMP width?
• Is the problem isolated to one job, node, rail, rack, or plane?

The network dashboard and the AI workload dashboard should share common identifiers for job, node, rail, port, switch, and fabric plane.

Design Failure Domains Before Production

A highly available fabric is not created by adding duplicate hardware without considering common dependencies.

Document the following failure domains:

• Individual cable.
• Transceiver.
• Adapter port.
• SuperNIC.
• Leaf switch.
• Spine switch.
• Rail.
• Fabric plane.
• Rack.
• Power distribution unit.
• Switch operating-system release.
• Configuration template.
• Automation pipeline.
• Management network.
• Monitoring platform.

A dual-plane network can still have a shared failure if both planes receive the same invalid configuration from one pipeline.

Similarly, redundant switches may share:

• The same rack power.
• The same cable path.
• The same console server.
• The same maintenance window.
• The same software defect.
• The same incorrect QoS profile.

Physical redundancy and operational independence must be evaluated separately.

Create a Drain-First Maintenance Model

Switch maintenance should begin by reducing production dependency on the device, not by immediately rebooting it.

A practical sequence is:

1. Identify active jobs and their fabric paths.
2. Confirm that alternate paths and switches are healthy.
3. Place the switch or interfaces into the approved maintenance or drain state.
4. Confirm that BGP routes withdraw as expected.
5. Verify the remaining ECMP width.
6. Observe traffic redistribution.
7. Perform the maintenance activity.
8. Validate hardware, interfaces, MTU, BGP, QoS, and telemetry.
9. Reintroduce the device gradually.
10. Confirm balanced traffic before closing the change.

Useful state checks include:

nv show maintenance
nv show interface
nv show router bgp
nv show qos
nv show router adaptive-routing

Maintenance testing must happen before the fabric is fully occupied. Discovering that route withdrawal or workload redistribution behaves differently than expected during a production upgrade is too late.

Implement the Fabric in Controlled Phases

A Spectrum-X deployment should progress through repeatable scale units.

Phase	Primary objective	Required exit criteria
Requirements	Define workload, scale, performance, separation, and availability	Approved architecture assumptions and success criteria
Bill of materials	Validate switches, SuperNICs, cables, optics, power, cooling, and software	Complete compatible and supportable component matrix
Physical build	Install racks, switches, management, cabling, and labels	Layer 1 validation passed with no unexplained errors
Underlay	Deploy BGP, routing, and ECMP	All adjacencies, routes, and path counts correct
RoCE baseline	Apply QoS, ECN, PFC, MTU, and host configuration	End-to-end RDMA connectivity and counter validation
Scale-unit pilot	Run representative collective and storage traffic	Performance, congestion, and failure tests passed
Failure testing	Remove links, leaves, spines, rails, and planes	Documented connectivity and degraded-performance objectives met
Operational acceptance	Validate NetQ, telemetry, alerts, maintenance, and rollback	Operations team can detect, isolate, drain, repair, and restore
Production scale	Repeat the validated unit	No undocumented variation from the approved baseline

A scale-unit approach limits the number of variables introduced at one time. It also creates a known-good reference that can be compared against later racks.

Validate the Fabric Under Realistic AI Traffic

Basic ping, BGP state, and interface counters are necessary, but they are not an AI-fabric acceptance test.

A useful test plan should include:

Connectivity Tests

• Every compute interface reaches its intended peers.
• Every rail follows the expected topology.
• All intended ECMP paths are installed.
• Large packets traverse without fragmentation or drops.
• RoCE communication succeeds across all leaf and spine combinations.

Bandwidth Tests

• Single-flow throughput.
• Multi-flow throughput.
• Node-to-node throughput.
• Rail-level throughput.
• Aggregate leaf throughput.
• Spine-path balance.
• Bidirectional traffic.
• Many-to-one and one-to-many patterns.

Collective Communication Tests

Test representative collective operations at the intended node count.

Measure:

• Per-operation duration.
• Variance between runs.
• Scaling efficiency.
• Slowest rail.
• Slowest node.
• GPU idle time.
• Network retransmission or congestion signals.
• Path utilization.

Congestion Tests

Create controlled contention and verify:

• ECN marking occurs before persistent loss.
• Sender behavior responds to congestion notification.
• PFC remains limited to the expected priority.
• Pause behavior does not spread unexpectedly.
• Adaptive routing uses available alternative paths.
• Receiver-side incast remains within defined performance limits.

Failure Tests

Remove one component at a time:

• Host-facing link.
• Spine-facing link.
• Leaf switch.
• Spine switch.
• Rail.
• Fabric plane.
• Management dependency.
• Telemetry collector.

Record both connectivity and performance outcomes.

Common Failure Patterns

Miswired Rails

Symptom: One rail shows consistently different path lengths, utilization, or performance.

Likely cause: Adapter positions do not map consistently to the intended leaf groups.

Validation: Compare cable records, LLDP neighbors, server interface identities, and switch descriptions.

Inconsistent MTU

Symptom: Small packets succeed, but large transfers or RoCE tests fail intermittently.

Likely cause: One host, adapter, switch port, or routed boundary uses a smaller effective MTU.

Validation: Test large nonfragmented packets across every rail and inspect drop counters.

QoS Marking Mismatch

Symptom: RoCE traffic reaches the wrong queue, ECN is absent, or PFC appears on an unexpected class.

Likely cause: Host marking, adapter trust, switch classification, or rewrite behavior does not match the design.

Validation: Trace the priority and traffic-class mapping from sender through receiver.

Excessive PFC Activity

Symptom: Pause frames remain elevated and congestion spreads across multiple links.

Likely cause: Persistent receiver congestion, insufficient source-rate control, an incorrect threshold, or PFC enabled too broadly.

Validation: Correlate PFC duration with ECN, CNP, queue occupancy, destination traffic, and workload timing.

Uneven ECMP Utilization

Symptom: One spine path is saturated while another remains lightly loaded.

Likely cause: Low-flow entropy, hash collisions, missing paths, or adaptive routing not operating as expected.

Validation: Confirm route count, ECMP width, per-link traffic, adaptive-routing state, and endpoint flow patterns.

Layer 1 Errors Under Load

Symptom: Links remain up but accumulate FEC, lane, symbol, or receive errors during sustained traffic.

Likely cause: Marginal optic, cable, connector, lane mapping, contamination, reach, or thermal condition.

Validation: Compare error rates under idle and load, then replace one physical component at a time.

Version Drift

Symptom: Supported commands, defaults, telemetry, or congestion behavior differ between switches.

Likely cause: Cumulus Linux, firmware, adapter software, or validated-profile versions are inconsistent.

Validation: Compare every component against the approved version matrix and configuration baseline.

Production Acceptance Checklist

Before scheduling production workloads, confirm:

• Compute, storage, in-band, and out-of-band boundaries are documented.
• The blocking ratio is calculated for normal and degraded conditions.
• Every leaf has the expected spine adjacencies.
• ECMP width matches the topology.
• Rails are consistently cabled and labeled.
• Spectrum switch, SuperNIC, firmware, and Cumulus Linux versions match the approved baseline.
• RoCE classification is verified end to end.
• ECN and congestion notification operate under controlled congestion.
• PFC is limited to the intended priority.
• MTU is validated across every path.
• Cabling and optics have no unexplained physical errors.
• Adaptive routing is enabled only on supported interfaces and speeds.
• NetQ receives the expected state and topology information.
• High-frequency telemetry exposes buffer, ECN, PFC, and path behavior.
• Failure tests meet documented connectivity objectives.
• Degraded performance is measured and accepted.
• Maintenance draining and restoration have been tested.
• Configuration rollback has been tested.
• The network and AI platform teams share escalation and ownership procedures.

Conclusion

Building an NVIDIA Spectrum-X Ethernet fabric requires more than installing high-speed switches and enabling RoCE.

The network must be designed around the behavior of distributed GPU workloads. That means separating unrelated traffic, calculating bisection bandwidth, aligning compute interfaces into rails, building a routed BGP underlay, and treating congestion control as an end-to-end system.

ECN, selective PFC, sender congestion control, ECMP, and adaptive routing each solve a different part of the problem. None of them compensates for an inconsistent MTU, a miswired rail, an unhealthy optic, or an oversubscribed leaf that was never tested under realistic traffic.

The strongest implementation pattern is to build one validated scale unit, prove it under collective communication and failure conditions, and then reproduce it through controlled automation. NetQ and high-frequency telemetry should be part of that scale unit from the beginning, not added after the first performance incident.

A successful AI factory fabric is not merely fast when every component is healthy. It remains understandable during congestion, predictable during failure, and maintainable while expensive GPU workloads continue to depend on it.

External References

• NVIDIA: Networking Logical Architecture – NVIDIA HGX AI Factory
  Canonical URL: https://docs.nvidia.com/enterprise-reference-architectures/hgx-ai-factory/latest/network-logical-architecture.html
• NVIDIA: Networking Physical Topologies – NVIDIA HGX AI Factory
  Canonical URL: https://docs.nvidia.com/enterprise-reference-architectures/hgx-ai-factory/latest/networking-physical-topologies.html
• NVIDIA: Network Fabrics – NVIDIA SuperPOD DGX B300 Systems Reference Architecture
  Canonical URL: https://docs.nvidia.com/dgx-superpod/reference-architecture/scalable-infrastructure-b300/latest/network-fabrics.html
• NVIDIA: Components – NVIDIA RTX PRO AI Factory
  Canonical URL: https://docs.nvidia.com/enterprise-reference-architectures/rtx-pro-ai-factory/latest/components.html
• NVIDIA: RDMA over Converged Ethernet – RoCE – Cumulus Linux 5.17
  Canonical URL: https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-517/Layer-1-and-Switch-Ports/Quality-of-Service/RDMA-over-Converged-Ethernet-RoCE/
• NVIDIA: Equal Cost Multipath Load Sharing – Cumulus Linux 5.17
  Canonical URL: https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-517/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing/
• NVIDIA: Optional BGP Configuration – Cumulus Linux 5.17
  Canonical URL: https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-517/Layer-3/Border-Gateway-Protocol-BGP/Optional-BGP-Configuration/
• NVIDIA: Switch Port Attributes – Cumulus Linux 5.17
  Canonical URL: https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-517/Layer-1-and-Switch-Ports/Interface-Configuration-and-Management/Switch-Port-Attributes/
• NVIDIA: NVIDIA User Experience – NVUE – Cumulus Linux 5.17
  Canonical URL: https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-517/System-Configuration/NVIDIA-User-Experience-NVUE/
• NVIDIA: Maintenance Mode – Cumulus Linux 5.17
  Canonical URL: https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-517/System-Configuration/Maintenance-Mode/
• NVIDIA: NVIDIA NetQ User Guide
  Canonical URL: https://docs.nvidia.com/networking-ethernet-software/cumulus-netq/
• NVIDIA: Open Telemetry Export – Cumulus Linux 5.17
  Canonical URL: https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-517/Monitoring-and-Troubleshooting/Open-Telemetry-Export/
• NVIDIA: High Frequency Telemetry – Cumulus Linux 5.17
  Canonical URL: https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-517/Monitoring-and-Troubleshooting/High-Frequency-Telemetry/
• NVIDIA: Mellanox What Just Happened – Cumulus Linux 5.17
  Canonical URL: https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-517/Monitoring-and-Troubleshooting/Network-Troubleshooting/Mellanox-WJH/
• NVIDIA: Introduction – NVIDIA Spectrum-4 SN5000 2U Switch Systems Hardware User Manual
  Canonical URL: https://networking-docs.nvidia.com/sn5000hw/introduction
• NVIDIA Technical Blog: Powering Next-Generation AI Networking with NVIDIA SuperNICs
  Canonical URL: https://developer.nvidia.com/blog/powering-next-generation-ai-networking-with-nvidia-supernics/
• NVIDIA Technical Blog: Turbocharging AI Workloads with NVIDIA Spectrum-X Networking Platform
  Canonical URL: https://developer.nvidia.com/blog/turbocharging-ai-workloads-with-nvidia-spectrum-x-networking-platform/
• NVIDIA Technical Blog: Next-Generation AI Factory Telemetry with NVIDIA Spectrum-X Ethernet
  Canonical URL: https://developer.nvidia.com/blog/next-generation-ai-factory-telemetry-with-nvidia-spectrum-x-ethernet/

The post How to Build an NVIDIA Spectrum-X Ethernet Fabric for an AI Factory appeared first on Digital Thought Disruption.