Megatron-LM's 3D parallelism strategy combines tensor parallelism, pipeline parallelism, and data parallelism. When scaling to thousands of GPUs, which placement of the parallelism dimensions relative to the hardware interconnect is preferred, and why?
- AData parallelism on the fastest intra-node links, pipeline parallelism in the middle, tensor parallelism across the slowest links; because gradient all-reduce tolerates high-latency inter-node links better than tensor all-reduce.
- BTensor parallelism across NVLink-connected GPUs within a node, pipeline parallelism across nodes, data parallelism across pipeline replicas; because tensor all-reduce is most frequent and requires the highest bandwidth. Correct
- CPipeline parallelism on the fastest intra-node links, tensor parallelism across nodes, data parallelism outermost; because pipeline bubbles are minimised when pipeline stages share fast NVLink bandwidth.
- DAll three parallelism dimensions applied uniformly across GPUs regardless of physical topology, because modern data-centre networks provide sufficient bandwidth at all levels to avoid bottlenecks.
Why A is wrong: This placement is inverted from recommended practice. Tensor parallelism requires high-bandwidth, low-latency all-reduce within each layer and must be placed on GPUs connected by fast NVLink. Placing it across the slowest interconnects would bottleneck training severely.
Why B is correct: Tensor parallelism synchronises partial activations at every transformer sub-layer (multiple all-reduces per layer per forward pass), so it needs the highest bandwidth and lowest latency available - intra-node NVLink. Pipeline parallelism communicates only at layer-group boundaries and tolerates inter-node links. Data parallelism's gradient all-reduce happens once per global step and can tolerate even slower interconnects, making it the outermost dimension.
Why C is wrong: Pipeline parallelism communicates at the boundaries of layer groups, not within individual layers, so it does not require as high a bandwidth as tensor parallelism. Placing tensor parallelism across inter-node links would route its high-frequency all-reduces over slow interconnects, a worse bottleneck than pipeline bubbles.
Why D is wrong: Physical network topology matters significantly at scale: NVLink within a node provides hundreds of gigabytes per second, while inter-node networking provides far less. Ignoring this hierarchy and applying tensor parallelism across slow inter-node links would create all-reduce bottlenecks that dominate training time.