NCA-AIIO - AI Infrastructure (40% of the exam) - Section 2.2
Scale a GPU infrastructure for different use cases.
Describe how GPU infrastructure scales from a single workstation through multi-GPU servers to multi-node clusters, and the role of NVLink, NVSwitch, and high-speed fabrics in enabling that progression. Choose the appropriate scale - single node, DGX BasePOD, or DGX SuperPOD - based on the compute and memory requirements of a given workload.
Practice question for this objective
lock_openFree sampleAI Infrastructuremedium
A server vendor builds an eight-GPU training node where every GPU must reach every other GPU at full point-to-point NVLink bandwidth simultaneously, not just its directly cabled neighbours. Direct GPU-to-GPU NVLink wiring alone cannot give all pairs full bandwidth at once. Which component provides the non-blocking all-to-all switching needed inside the node?
AA PCIe Gen5 switch that fans the GPUs out to the host root complex and arbitrates their peer-to-peer transfers
BA BlueField DPU that terminates the NVLink fabric and forwards GPU memory transactions through its embedded Arm cores
CAn InfiniBand HDR switch placed inside the chassis to carry the GPU-to-GPU collective traffic between the eight GPUs
DAn NVSwitch fabric that cross-connects every GPU's NVLink ports so any pair communicates at full NVLink bandwidth concurrentlycheck_circle Correct
NVSwitch provides non-blocking all-to-all switching of NVLink so every GPU pair in a node communicates at full bandwidth simultaneously. Direct NVLink only wires each GPU to a limited set of peers, so simultaneous all-pairs traffic would contend for links. NVSwitch is a hardware crossbar that connects every GPU's NVLink ports through a switched fabric, letting any GPU reach any other at full NVLink bandwidth without going through PCIe or the CPU.
Why A is wrong: Tempting because a PCIe switch does interconnect devices, but PCIe peer-to-peer is far slower than NVLink and routing GPU traffic through the host complex is exactly the bottleneck NVLink exists to bypass.
Why B is wrong: The DPU is genuine hardware, but it offloads networking, storage, and security at the host boundary; it does not switch intra-node NVLink traffic, so it cannot deliver all-to-all GPU bandwidth.
Why C is wrong: InfiniBand is the scale-out fabric between nodes, not an intra-node memory interconnect; routing eight on-board GPUs through an InfiniBand switch would add NIC and protocol latency that NVSwitch avoids entirely.
Why D is correct: Correct: NVSwitch is the on-board crossbar that interconnects all GPUs' NVLink lanes, giving non-blocking all-to-all connectivity so every GPU pair exchanges data at full NVLink bandwidth at the same time.
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-AIIO bank for this domain.
Replicate the full model on each GPU using data parallelism and shard only the input batches across the GPUs.
errorWhy it is wrong: Data parallelism is the most familiar scaling pattern, so it is tempting, but it requires the entire model, including all 600 GB of embeddings, to fit on every single GPU, which is exactly the constraint that has been exceeded.
NVSwitch compresses the gradient tensors in flight, so the same physical links carry more useful payload during the collective.
errorWhy it is wrong: Tempting because in-network reduction such as SHARP does cut traffic on some fabrics, but NVSwitch does not compress tensor payloads; it switches NVLink traffic, and compression is not the reason all-to-all bandwidth holds up.
Gradient accumulation, which splits the batch into smaller micro-batches and sums their gradients before one optimiser step.
errorWhy it is wrong: This is tempting because gradient accumulation also eases memory pressure, but it is wrong because it works by shrinking the per-step micro-batch; the scenario requires keeping the same batch size, and accumulation does not reduce the activation memory of a given micro-batch.