A team is preparing to train a large language model whose total memory footprint - accounting for model parameters, intermediate activations, and optimiser states - comfortably exceeds the capacity of a single GPU. Which infrastructure decision most directly addresses this constraint?
- AReplace the GPUs with DPUs to offload memory management to a dedicated data-processing unit.
- BSwitch to a higher-throughput Ethernet fabric between nodes, as network latency is the bottleneck that prevents the model fitting in memory.
- CIncrease fast NVMe storage capacity so the model can be streamed from disk into GPU memory in chunks during the forward pass.
- DDistribute the model across multiple GPUs so the combined memory across the pool can hold all training states simultaneously. Correct
Why A is wrong: DPUs handle networking, storage, and security offload from the CPU; they have no general-purpose tensor compute memory that substitutes for GPU VRAM in training workloads.
Why B is wrong: Network throughput affects gradient synchronisation speed but does not change how much GPU memory is available; upgrading the fabric cannot make a model fit in memory it does not have.
Why C is wrong: While CPU offload and disk-backed parameter swapping exist as workarounds, they introduce severe performance penalties; the standard architectural decision is to add GPUs until combined memory is sufficient, not to rely on disk streaming.
Why D is correct: When a model's parameters, activations, and optimiser states together exceed one GPU's memory, spreading them across multiple GPUs is the standard solution - each GPU holds a partition of the total working set.