NCA-AIIO - AI Infrastructure (40% of the exam) - Section 2.1
Identify hardware requirements for specific AI training task use cases.
Identify the GPU memory, interconnect bandwidth, and storage throughput requirements driven by model size, batch size, and dataset scale for common AI training workloads. Distinguish why large language model training demands high-bandwidth memory (HBM) GPUs and fast NVLink or InfiniBand interconnects compared with smaller computer-vision tasks.
Practice question for this objective
lock_openFree sampleAI Infrastructuremedium
A research team is training a large language model where the model weights alone exceed the memory capacity of a single GPU. The team wants to minimise inter-node communication overhead. Which scaling approach and parallelism strategy is the most appropriate combination?
AScale-out across many nodes via InfiniBand fabric, using data parallelism to replicate the full model on every node.
BScale-out across many nodes via Ethernet fabric, applying pipeline parallelism to overlap computation across nodes.
CScale-up within a single node using NVLink/NVSwitch high-bandwidth interconnect, applying model parallelism to partition the model across GPUs.check_circle Correct
DScale-up within a single node using NVLink/NVSwitch, applying data parallelism to shard the training dataset across GPUs.
Identify when scale-up with model parallelism is the correct approach for memory-bound large model training. Model parallelism partitions the model itself (by layers in pipeline parallelism, or by tensors in tensor parallelism) across multiple GPUs so that no single GPU needs to store all weights. Performing this within a multi-GPU node connected by NVLink/NVSwitch leverages the high aggregate bandwidth of that interconnect, minimising the communication cost that model parallelism inherently incurs relative to crossing an inter-node network.
Why A is wrong: Data parallelism replicates the entire model on each accelerator, so it cannot address a model that exceeds single-GPU memory capacity; scale-out alone does not solve the memory constraint.
Why B is wrong: Pipeline parallelism can be used across nodes, but Ethernet fabric introduces higher latency and lower bandwidth than NVLink or InfiniBand for the tight synchronisation between pipeline stages, making it a poor fit when minimising communication overhead is the goal.
Why C is correct: When model weights exceed one GPU's memory, model parallelism splits layers or tensors across GPUs. Doing this within a node over NVLink/NVSwitch maximises bandwidth and minimises latency compared with crossing an inter-node fabric.
Why D is wrong: Data parallelism shards data, not model weights, so each GPU still needs to hold the full model in memory. This does not resolve the original constraint of model weights exceeding a single GPU's memory.
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-AIIO bank for this domain.
Provision a single shared NIC per server regardless of GPU count, since collective traffic is bursty and rarely uses more than one adapter at a time.
errorWhy it is wrong: Tempting because one NIC simplifies cabling, but synchronous all-reduce drives every GPU to send concurrently, so a single shared adapter becomes the node-boundary bottleneck that throttles the whole step.
It routes inter-GPU traffic across the data centre spine so that any two GPUs anywhere in the cluster can communicate directly
errorWhy it is wrong: It is tempting because scale-out fabrics do connect GPUs across the cluster, but NVLink is a short-reach intra-node and intra-rack interconnect, not a cluster-wide spine fabric, so this misstates its role.
Add more GPUs per node so the additional compute hides the time spent waiting for each input batch to arrive.
errorWhy it is wrong: This is tempting because more GPUs sounds like more capability, but it is wrong because the job is input-bound, not compute-bound; adding GPUs raises the aggregate read demand on the already-saturated filesystem and worsens the stall.