A research team is training a large language model where the model weights alone exceed the memory capacity of a single GPU. The team wants to minimise inter-node communication overhead. Which scaling approach and parallelism strategy is the most appropriate combination?
- AScale-out across many nodes via InfiniBand fabric, using data parallelism to replicate the full model on every node.
- BScale-out across many nodes via Ethernet fabric, applying pipeline parallelism to overlap computation across nodes.
- CScale-up within a single node using NVLink/NVSwitch high-bandwidth interconnect, applying model parallelism to partition the model across GPUs. Correct
- DScale-up within a single node using NVLink/NVSwitch, applying data parallelism to shard the training dataset across GPUs.
Why A is wrong: Data parallelism replicates the entire model on each accelerator, so it cannot address a model that exceeds single-GPU memory capacity; scale-out alone does not solve the memory constraint.
Why B is wrong: Pipeline parallelism can be used across nodes, but Ethernet fabric introduces higher latency and lower bandwidth than NVLink or InfiniBand for the tight synchronisation between pipeline stages, making it a poor fit when minimising communication overhead is the goal.
Why C is correct: When model weights exceed one GPU's memory, model parallelism splits layers or tensors across GPUs. Doing this within a node over NVLink/NVSwitch maximises bandwidth and minimises latency compared with crossing an inter-node fabric.
Why D is wrong: Data parallelism shards data, not model weights, so each GPU still needs to hold the full model in memory. This does not resolve the original constraint of model weights exceeding a single GPU's memory.