TL;DR

An NVIDIA NVL72 AI factory is not a collection of GPU servers connected to a fast Ethernet network. Each rack is a tightly coupled, 72-GPU NVLink scale-up domain with rack-level firmware, power, cooling, management, and service dependencies. Additional racks are joined through a separate scale-out fabric, commonly a dual-plane, rail-optimized Spectrum-X design for current GB300 reference architectures, or a validated InfiniBand design in applicable GB200 deployments.

The practical design rule is simple: treat the rack as the atomic compute system, treat the fabric as a separate distributed system, and validate both through an evidence chain that starts with facilities and firmware and ends with NCCL, storage, inference, resilience, and operational handover. NVIDIA certification can narrow procurement risk, but it does not replace site-specific architecture review or cluster acceptance testing.

Introduction

The hardest part of an NVIDIA GB200 or GB300 NVL72 deployment is not selecting a GPU. It is making a rack-scale computer behave like a dependable enterprise platform inside a real data center.

That distinction matters because an NVL72 rack compresses several normally separate design problems into one operational unit. Compute trays, NVLink switch trays, DPUs, network adapters, rack power shelves, liquid cooling, firmware, Fabric Manager, telemetry, orchestration, storage, and the scale-out network must all agree. A weakness in any one layer can prevent the cluster from reaching the performance, availability, or maintainability expected from the hardware investment.

A successful design therefore begins below Kubernetes and below the leaf-spine fabric. It begins with ownership, facility readiness, supported component recipes, and a clear understanding of where the scale-up domain ends and the scale-out fabric begins.

This deep dive provides a practical architecture and validation framework for teams designing GB200 NVL72 or GB300 NVL72 AI factories. It does not prescribe one universal bill of materials. OEM implementations, validated designs, software releases, and site constraints still govern the final solution.

Design the AI Factory as a System, Not a Rack Purchase

The most important architecture decision is the unit of design.

Traditional server clusters often treat the node as the smallest independently deployable and serviceable component. An NVL72 system changes that model. The rack contains 18 compute trays and nine NVLink switch trays that form one 72-GPU NVLink domain. The rack also depends on coordinated power, cooling, management, and software services. A compute tray is replaceable, but it is not an isolated four-GPU island from the perspective of the complete platform.

The design should use three nested units:

1. The tray is a hardware service unit.
2. The rack is the scale-up compute and NVLink failure domain.
3. The cluster is the scale-out scheduling, storage, and network domain.

This framing prevents a common design mistake: treating all high-speed links as one undifferentiated fabric.

The architecture is not complete until each unit has an owner, an acceptance gate, and a maintenance procedure.

Establish Rack-Scale Ownership Boundaries

NVL72 projects often fail organizationally before they fail technically. The infrastructure crosses facilities, networking, platform engineering, storage, security, AI operations, and vendor support. When responsibility is vague, design gaps surface during commissioning, when changes are expensive and schedule pressure is high.

A practical ownership model should assign accountability before hardware arrives.

The ownership boundary between the OEM and the customer deserves special attention. A rack can arrive factory-integrated and still require customer-owned design decisions for fabric topology, IP addressing, storage, control-plane placement, tenant isolation, operational tooling, and workload validation.

Understand the NVL72 Scale-Up Domain

An NVL72 rack contains 18 compute trays. Each tray contributes four Blackwell-family GPUs and two Grace CPUs. Nine NVLink switch trays provide the rack-scale switching layer that joins all 72 GPUs into one NVLink domain.

That topology is the defining property of the system. It allows workloads to use the rack as a tightly coupled accelerator complex, but it also creates dependencies that do not exist in loosely coupled PCIe GPU servers. Fabric Manager, NVSwitch health, rack-level firmware compatibility, power sequencing, and topology discovery become first-class platform requirements.

The following diagram shows the architecture boundary that should drive the rest of the design.

The reader should notice two separate communication systems. NVLink and NVSwitch create the scale-up domain inside the rack. Spectrum-X or InfiniBand creates the scale-out domain between racks. They solve different problems and have different owners, telemetry, failure modes, and acceptance tests.

GB200 and GB300 Are Related, Not Interchangeable

The design method is common, but the component details are not identical.

Power figures are planning anchors, not construction values. The final facility design must use the selected OEM rack, exact power shelves, configured components, redundancy mode, regional electrical requirements, and measured commissioning data.

Separate Scale-Up from Scale-Out Networking

Scale-up and scale-out are often discussed as if they are only bandwidth tiers. The more useful distinction is the communication behavior they support.

NVLink is optimized for tightly coupled GPU communication inside the rack. It allows the 72 GPUs to participate in a common high-bandwidth domain through NVSwitch. The scale-out network connects separate NVLink domains and external services. It carries inter-rack collectives, checkpoint traffic, storage access, orchestration traffic, telemetry, and service communication according to the detailed design.

This separation changes several architecture decisions:

• A healthy Ethernet or InfiniBand fabric does not prove that the NVLink domain is healthy.
• A healthy NVLink domain does not prove that multi-rack NCCL will perform correctly.
• A per-node network benchmark cannot prove that rail mapping is correct across the cluster.
• A successful synthetic collective does not prove that storage or inference SLOs are met.
• Maintenance inside one rack can affect a 72-GPU scheduling unit even when the scale-out network remains available.

The acceptance plan must test each boundary independently and then test the complete path as one system.

Build a Dual-Plane, Rail-Optimized Spectrum-X Fabric

Current NVIDIA GB300 reference material uses an RDMA-based, full nonblocking leaf-spine compute fabric with rail-optimized GPU connectivity. The reference design separates the scale-out network into two independent planes.

A rail is the repeated network path associated with a consistent GPU position across compute trays. Instead of connecting ports arbitrarily to any leaf, the design maps equivalent GPU-facing ports to the same logical rail. This gives collective communication software a predictable topology and reduces the chance that traffic concentration or asymmetric paths undermine performance.

A dual-plane design gives each GPU-facing path access to two independently routed fabrics. It is not simply two VLANs on one physical network. The planes should have independent links and switching paths, with the degree of physical separation defined by the resilience objective.

Fabric Design Criteria

The detailed design should explicitly document:

• Nonblocking target: Define whether the compute fabric is truly 1:1 at the intended cluster scale, including any super-spine tier.
• Plane independence: Identify shared power, shared racks, shared optics paths, and shared management dependencies that could defeat the apparent redundancy.
• Rail mapping: Maintain a port-to-GPU-to-leaf map that survives installation, replacement, and expansion.
• RDMA configuration: Standardize RoCE, priority flow control where required, ECN, queue design, MTU, DSCP, and congestion-control behavior according to the validated recipe.
• Adaptive routing and load balancing: Confirm the selected switch and NIC software features are supported together and measure their behavior under collective workloads.
• Telemetry: Collect interface counters, congestion signals, buffer events, FEC, optical health, RDMA counters, retransmissions, and flow-level evidence.
• Cabling discipline: Use a machine-readable cable plan, unique labels, optical budgets, inspection procedures, and post-install link validation.
• Expansion headroom: Reserve leaf, spine, super-spine, optics, IP, and power capacity for the next planned rack unit.

The network team should be able to prove the mapping from a logical NCCL rail to a physical GPU port, cable, switch port, plane, and spine path. A diagram without this operational mapping is not enough.

Keep Compute, Storage, Customer, and Management Fabrics Distinct

A practical NVL72 design usually contains several networks even when some functions share physical switches. The architecture should describe the functions separately before deciding which can be converged.

Convergence can be valid, but it must be a decision rather than an assumption. Sharing a physical fabric between compute, storage, and customer traffic changes congestion control, security ownership, blast radius, and acceptance testing. The design should state what is shared, what is isolated, and what happens when one traffic class becomes abusive.

Treat Power, Cooling, and Rack Weight as Architecture Inputs

A high-density NVL72 rack is a facility project and a computing project at the same time. The cluster design is not ready when the logical topology is complete. It is ready when the data center can deliver and sustain the required electrical, thermal, structural, and service conditions.

Facility Readiness Checklist

Rack weight must be taken from the exact selected system, not a generic NVL72 label. Integrated rack weight changes with the OEM frame, power shelves, manifolds, switches, cabling, shipping configuration, and optional components. The floor-load study should include the delivery route and point loads, not only the final rack footprint.

The power design should also distinguish nameplate capacity from measured operating demand. Capacity planning needs enough headroom for workload peaks, component aging, redundancy, and future software behavior. It should not assume that an average training run represents the worst electrical state.

Make Fabric Manager and Rack Software Dependencies Explicit

The NVLink domain depends on a coordinated software stack. Fabric Manager is not an optional monitoring add-on. It participates in discovering, configuring, routing, and monitoring the NVLink and NVSwitch fabric, while coordinating with GPU drivers and the broader rack software environment.

The design should document at least these dependencies:

• NVSwitch firmware and switch operating software
• Fabric Manager and NVLink Subnet Manager functions
• GPU driver and CUDA compatibility
• NVIDIA Collective Communications Library
• Inter-Memory Exchange services where used
• Data Center GPU Manager health and telemetry
• Network operating systems, NIC and DPU firmware, and RDMA software
• Mission Control or the selected rack-management and lifecycle platform
• Baseboard management, rack power, cooling, and leak telemetry

The names are easy to confuse. Fabric Manager manages the NVLink fabric. Ethernet fabric operations use the selected Spectrum-X management and telemetry tooling. InfiniBand deployments use the appropriate InfiniBand fabric-management stack. These systems can be integrated operationally, but they are not interchangeable.

Power-On and Restart Sequencing

Rack power sequencing is an architecture dependency because the NVLink switch layer must be available for the compute nodes to discover and join the intended domain. The operational runbook should specify the supported order for power-on, shutdown, and recovery. It should also define when compute nodes must be restarted after a switch-layer or management-layer event.

A generic data-center instruction such as “power on the rack” is insufficient. The procedure should include:

1. Facility and cooling readiness confirmation.
2. Rack power and management verification.
3. NVLink switch-layer initialization.
4. Fabric Manager and topology confirmation.
5. Compute-tray initialization.
6. GPU, NVLink, NIC, DPU, and storage health checks.
7. Scheduler admission only after the rack passes its gate.

Treat the Release Recipe as One Configuration Item

The supported state is a recipe, not a collection of independently latest packages. GPU drivers, CUDA, Fabric Manager, NCCL, DCGM, switch firmware, NIC firmware, DPU software, DGX OS or the selected host OS, and cluster operators must be tested as a combination.

At the July 25, 2026 research cutoff, NVIDIA had posted GB200 rack release 1.3.10 and GB300 rack release 1.0.10, both dated July 15, 2026. The release notes also identify minimum rack-wide release levels for NVLink Recovery compatibility. Those values are useful examples of why lifecycle management must be rack-wide, but they should be rechecked immediately before publication, procurement, or upgrade execution.

Layer Kubernetes and Cluster Software Above a Stable Rack Baseline

Kubernetes does not remove the need for rack-level validation. It adds another control plane above the hardware and low-level software stack. The cluster should not admit the NVL72 rack until the rack has already passed hardware, firmware, NVLink, network, and storage gates.

The reader should notice that Kubernetes is near the top of the dependency chain. It can schedule a resource only after lower layers expose a truthful and stable resource state.

Separate Administration from Workload Tenancy

For larger AI factories, use a dedicated management or administration cluster for lifecycle services, deployment automation, images, telemetry, and platform control functions. Avoid placing critical rack-management services exclusively inside the same user cluster they must recover.

The user cluster should enforce:

• homogeneous node pools for the relevant GB200 or GB300 configuration
• explicit GPU and topology labels
• topology-aware scheduling and placement
• taints, tolerations, and admission policies
• controlled access to RDMA devices and host capabilities
• version-pinned GPU and Network Operators
• namespace, quota, and priority controls
• maintenance drain and return-to-service gates
• centralized logs, metrics, fabric telemetry, and audit trails

The current Network Operator and GPU Operator support matrices should be treated as release inputs. Kubernetes, operating-system, driver, OFED, NIC, and operator compatibility changes over time. Copying an operator version from a lab build into production without checking the complete matrix is a preventable failure.

Design Failure Domains Before You Schedule Jobs

A cluster is not resilient because it has redundant links. It is resilient when the scheduler, network, storage, facility systems, and operators agree on what can fail together and what action follows.

The expected behavior should be written as a testable hypothesis. For example, “Plane B preserves reachability after Plane A isolation” is a design objective, not a fact, until the exact installed topology and software recipe prove it under representative load.

Use Cluster Acceptance Testing as a Gated Evidence Chain

Acceptance testing should prevent an unhealthy layer from being hidden by a higher-level benchmark. A large language model can sometimes run on a partially degraded platform, but that does not mean the platform is ready for production.

A useful evidence chain moves from static configuration to dynamic workload behavior.

Use an Acceptance Manifest

A versioned acceptance manifest prevents the test team from validating one configuration while operations inherits another. The following YAML is a framework, not a vendor-supplied schema.

acceptance_baseline:
  platform: gb300-nvl72
  rack_count: 4
  hardware_bom: "<approved-bom-id>"
  software_recipe: "<approved-release-id>"

  fabric:
    type: spectrum-x
    planes: 2
    oversubscription: "1:1"
    rail_map: "<controlled-artifact-id>"

  required_tests:
    - facility_commissioning
    - firmware_inventory
    - dcgm_health
    - nvlink_domain
    - per_rail_rdma
    - nccl_collectives
    - gpudirect_storage
    - scheduler_drain_and_return
    - inference_slo
    - failure_injection
    - soak_test

  evidence:
    repository: "<artifact-repository>"
    retention_days: 365
    required_approvers:
      - facilities
      - network
      - storage
      - platform
      - ai_operations
      - security

The fields to change are the platform, rack count, approved BOM, release recipe, fabric type, rail-map artifact, test list, retention period, and approvers. Success means every test result can be traced to the installed hardware and software baseline. A common failure is retaining benchmark output without the topology, versions, workload parameters, or environmental state needed to interpret it.

Validate NCCL and Inference at the Topology Boundaries

NCCL testing is necessary because it exposes the interaction between GPUs, NVLink, RDMA, topology discovery, routing, and collective algorithms. It is not sufficient because production workloads add model, framework, storage, serving, and concurrency behavior.

Build a Layered NCCL Matrix

Run collective tests at multiple scopes:

Record more than the headline bandwidth number. Retain message sizes, process placement, GPU mapping, NCCL environment, topology files, algorithm selection, transport selection, CPU and NUMA affinity, rail counters, switch counters, optical health, retransmissions, and run-to-run variance.

Pass thresholds should come from the validated design, vendor or integrator baseline, and the exact installed bill of materials. An arbitrary value copied from a different cluster can hide topology differences or create a false failure.

Validate Inference as a Service, Not a Kernel

Inference acceptance should use representative models and serving behavior. At minimum, measure:

• model load and warm-up time
• time to first token
• inter-token latency
• request throughput
• tokens per second per user and across the service
• P50, P95, and P99 latency
• prefill and decode behavior
• concurrency and queueing
• HBM and KV-cache utilization
• storage and model-loading paths
• GPU, network, CPU, memory, and power utilization
• behavior during a failed node, link, plane, or service component
• recovery, rescheduling, and return to full capacity

The test should state the model, precision, framework, serving stack, tensor and pipeline parallelism, sequence lengths, input and output distributions, concurrency, batching, scheduler policy, and SLO. Without those details, an inference number is not reusable evidence.

For training workloads, apply the same principle. Record samples or tokens per second, scaling efficiency, checkpoint behavior, restart time, convergence or quality controls, and the infrastructure state during the run.

Expand Capacity in Scalable Units, Not Random Racks

Capacity expansion should preserve the original architecture rather than append hardware wherever space remains.

Before the first rack is installed, the design should reserve:

• rail-aligned leaf ports
• spine or super-spine capacity
• optics, patching, and cable pathways
• IP address, VLAN, routing, and management capacity
• rack power and upstream electrical capacity
• CDU, secondary-loop, and facility-water headroom
• storage throughput and metadata capacity
• management-cluster and telemetry capacity
• scheduler, quota, and queue capacity
• service access, spares, and maintenance space

Add racks symmetrically across the two network planes. Preserve the GPU-to-rail mapping. Update the source-of-truth inventory and cable map before installation. Then rerun the acceptance tests affected by the change, including per-rail RDMA, multi-rack NCCL, storage, workload, and failure tests.

Mixing GB200 and GB300 in one administrative estate may be operationally reasonable, but it should not imply one homogeneous scheduling pool. Different GPUs, NIC generations, power envelopes, firmware, drivers, operators, and performance profiles can complicate placement and support. Use separate node pools or clusters unless the exact mixed design and workload behavior have been validated.

Plan Maintenance and Lifecycle Around the NVLink Domain

A rack-scale system changes maintenance economics. A component replacement may be physically local but operationally rack-wide. The maintenance model should assume that some interventions require draining the entire NVL72 domain, preserving fault evidence, applying a coordinated recipe, and rerunning acceptance gates.

Define Maintenance Classes

Use Canary and Wave-Based Upgrades

Do not update all racks at once. Maintain an approved production recipe, a test or canary environment that is representative enough to expose compatibility problems, and a wave plan with explicit stop conditions.

A sound lifecycle workflow is:

1. Validate the candidate recipe in a nonproduction or canary rack.
2. Run the complete affected acceptance suite.
3. Drain one production rack.
4. Apply the change with captured before-and-after inventory.
5. Revalidate the rack and its fabric paths.
6. Return it to service under observation.
7. Continue only when the stop conditions remain clear.
8. Retain the previous recipe, rollback method, and evidence.

Firmware and software “latest” is not a lifecycle strategy. Supported compatibility and reproducible evidence are the strategy.

Follow a Practical Design and Validation Sequence

The work should progress through decision gates rather than a single installation project.

Discover and Scope

Define workloads, model sizes, training and inference patterns, data volumes, RTO and RPO, security boundaries, tenancy, growth, and operational ownership. Decide whether GB200 or GB300 is the correct platform and whether the scale-out design will use Spectrum-X Ethernet or a supported InfiniBand architecture.

Exit criterion: Approved requirements, assumptions, workload profiles, and ownership model.

Qualify the Facility

Complete electrical, cooling, structural, delivery-route, service-clearance, water, leak-detection, and emergency-procedure studies against the exact OEM rack.

Exit criterion: Signed facility design and capacity reservation for the initial and planned expansion phases.

Complete the Detailed Architecture

Produce logical and physical network designs, rail maps, plane boundaries, storage architecture, management and security zones, IP plans, control-plane placement, software recipe, failure domains, and observability design.

Exit criterion: A buildable, reviewable design with no unresolved ownership boundary.

Factory and Integration Validation

Confirm certified components, exact BOM, firmware inventory, cable plan, SBOM, factory test evidence, shipment controls, and on-site prerequisites.

Exit criterion: The delivered configuration matches the approved design and support baseline.

Rack Bring-Up

Commission cooling and power, initialize management and switch layers, establish Fabric Manager health, bring up compute trays, validate inventory, and confirm the 72-GPU NVLink topology.

Exit criterion: Each rack passes its standalone health and NVLink gate.

Fabric and Storage Acceptance

Validate rail mapping, both planes, RDMA, leaf-spine behavior, congestion controls, storage paths, GDS where required, and management reachability.

Exit criterion: Every rack has symmetric, supportable paths to the required cluster services.

Cluster and Workload Acceptance

Validate Kubernetes or Slurm, operators, scheduling, NCCL at increasing scale, representative training or inference, failure behavior, and soak stability.

Exit criterion: The platform meets documented technical and service SLOs with reproducible evidence.

Operational Handover

Deliver runbooks, dashboards, alerts, spares, support paths, escalation, maintenance classes, upgrade procedure, rollback, evidence retention, and capacity thresholds.

Exit criterion: Operations can detect, isolate, recover, maintain, and expand the platform without relying on undocumented project knowledge.

Conclusion

A GB200 or GB300 NVL72 AI factory should be designed from the facility upward and validated from the tray outward. The rack is a 72-GPU NVLink scale-up domain, not a loose collection of servers. The scale-out network is a separate distributed system that must preserve rail symmetry, plane independence, RDMA behavior, and predictable performance across racks.

The strongest design does not stop at a reference diagram. It assigns ownership, fixes a supported release recipe, makes failure domains explicit, preserves expansion headroom, and defines an acceptance evidence chain. Facilities, Fabric Manager, Spectrum-X or InfiniBand, storage, Kubernetes, NCCL, and the workload are all part of the same production outcome.

NVIDIA certification and reference architectures are valuable inputs. They reduce uncertainty about supported components and known patterns. They do not prove that a specific site, fabric, storage system, software recipe, or workload has been implemented correctly. That proof comes from disciplined commissioning, topology-aware testing, failure injection, workload validation, and repeatable lifecycle operations.

The practical goal is not merely to power on 72 GPUs. It is to build an AI factory that can be measured, maintained, expanded, and trusted under production conditions.

External References

• NVIDIA: NVIDIA NVL72 AI Factory
  Canonical URL: https://docs.nvidia.com/enterprise-reference-architectures/nvl72-ai-factory/latest/index.html
• NVIDIA: Network Logical Architecture
  Canonical URL: https://docs.nvidia.com/enterprise-reference-architectures/nvl72-ai-factory/latest/network-logical-architecture.html
• NVIDIA: System Hardware & Components
  Canonical URL: https://docs.nvidia.com/enterprise-reference-architectures/nvl72-ai-factory/latest/components.html
• NVIDIA: Hardware, NVIDIA DGX GB Rack Scale Systems User Guide
  Canonical URL: https://docs.nvidia.com/dgx/dgxgb200-user-guide/hardware.html
• NVIDIA: Software, NVIDIA DGX GB Rack Scale Systems User Guide
  Canonical URL: https://docs.nvidia.com/dgx/dgxgb200-user-guide/software.html
• NVIDIA: NVIDIA Mission Control
  Canonical URL: https://docs.nvidia.com/mission-control/index.html
• NVIDIA: NVIDIA Mission Control Management Plane and Rack Setup with NVIDIA GB200 or GB300 NVL72 Systems Installation Guide
  Canonical URL: https://docs.nvidia.com/mission-control/docs/rack-bring-up-install/index.html
• NVIDIA: Rack Bring-Up
  Canonical URL: https://docs.nvidia.com/mission-control/docs/rack-bring-up-install/2.3.0/rack-bring-up.html
• NVIDIA: Microbenchmarks, NVIDIA HPC Benchmarks
  Canonical URL: https://docs.nvidia.com/nvidia-hpc-benchmarks/Microbenchmarks.html
• NVIDIA: Inference Provider Platform Requirements on GB300 NVL72
  Canonical URL: https://docs.nvidia.com/dsx/ncp/inference-provider-requirements/home
• NVIDIA: Platform Support, NVIDIA Network Operator
  Canonical URL: https://docs.nvidia.com/networking/display/kubernetes2640/platform-support.html
• NVIDIA: NVIDIA-Certified Systems
  Canonical URL: https://docs.nvidia.com/certification-programs/latest/nvidia-certified-systems.html
• NVIDIA: NVIDIA DGX GB200 NVL72 Release Notes
  Canonical URL: https://docs.nvidia.com/dgx/dgxgb200nvl72-release-notes/dgxgb200nvl72-release-notes.pdf
• NVIDIA: NVIDIA DGX GB300 NVL72 Release Notes
  Canonical URL: https://docs.nvidia.com/dgx/dgxgb300nvl72-release-notes/dgxgb300nvl72-release-notes.pdf

The post How to Design and Validate an NVIDIA GB200 or GB300 NVL72 AI Factory appeared first on Digital Thought Disruption.