Free NCA-AIIO flashcards

When an LLM's training memory footprint (parameters, activations, optimiser states) exceeds a single GPU's capacity, what infrastructure decision most directly addresses it?

Distribute the model across multiple GPUs so the combined memory of the pool can hold all training states at once. Strategies such as tensor parallelism or pipeline parallelism partition the working set, with each GPU holding a portion. This is the standard solution when parameters, activations, and optimiser states together exceed one GPU.

If a model exceeds GPU memory, you can replace GPUs with DPUs to offload memory management to a dedicated data-processing unit.

DPUs handle networking, storage, and security offload from the CPU. They have no general-purpose tensor compute memory that substitutes for GPU VRAM in training, so they do not expand the GPU memory pool the training process needs.

What three core technical pillars are most widely credited for enabling the dramatic performance improvements in deep learning over the past decade?

The availability of large datasets (big data), GPU-based parallel compute whose massively parallel architecture suits matrix-heavy operations, and advances in model architectures such as the transformer introduced in 2017. Together these pillars let models scale in ways CPU-only, small-data, or older architectures could not support.

Wider adoption of NVLink interconnects inside GPU servers is what lets teams without infrastructure access large model training.

NVLink is a high-bandwidth GPU-to-GPU interconnect that improves multi-GPU throughput within a node. It is a hardware feature inside a server, not something that reduces entry barriers for teams with no on-premises GPUs at all.

Which scheduling strategy prevents partial-allocation failures in multi-node distributed GPU training, where a job stalls because only some of the required nodes are claimed before a competitor takes the rest?

Gang scheduling (co-scheduling) treats all processes of a distributed job as a single indivisible unit. It withholds the job until every required slot on every node is available simultaneously, then launches all ranks together. This atomic allocation eliminates the partial-allocation deadlock and is essential for tightly coupled frameworks like MPI or PyTorch distributed where all ranks must communicate from the first iteration.

Fair-share scheduling, which divides GPU resources proportionally among users, will prevent a multi-node job from failing due to partial node allocation.

Fair-share scheduling governs resource equity across users over time, but it does not guarantee that all nodes for a single job are reserved simultaneously, so the same partial-allocation race can still occur. Gang scheduling is the mechanism that co-allocates all nodes at once.