An architect is reviewing the network design for a new accelerated cluster. The plan places GPU collective traffic, parallel-storage reads and out-of-band management telemetry all on one shared fabric to simplify cabling. During trial runs the team sees gradient all-reduce throughput collapse whenever a large checkpoint is written. Which design principle should the architect apply to prevent this interference?
- ARaise the management network bandwidth so that telemetry no longer competes with the storage and compute traffic on the shared fabric.
- BSchedule all checkpoint writes to occur only at the very end of each training job so they never overlap with gradient exchange.
- CSwitch the shared fabric from InfiniBand to Ethernet so that congestion control on Ethernet automatically resolves the contention between flows.
- DSeparate the cluster onto purpose-built fabrics, giving GPU collective traffic its own dedicated compute fabric isolated from storage and management networks. Correct
Why A is wrong: It is tempting because more bandwidth sounds like it would relieve congestion, but the contention is between storage and compute flows, so enlarging the low-volume management network does not stop checkpoints from throttling all-reduce.
Why B is wrong: It is tempting because timing the writes avoids overlap, but periodic checkpointing during long runs is needed for fault tolerance, so deferring it sacrifices resilience and still leaves the fabrics coupled.
Why C is wrong: It is tempting because Ethernet has congestion-control features, but changing the link technology on a single shared fabric does not separate the competing flows, and the storage bursts would still degrade collectives.
Why D is correct: This is correct because reference accelerated-cluster designs segregate traffic by purpose so that bursty storage writes cannot steal bandwidth from latency-sensitive GPU collective operations.