A PyTorch training pipeline loads a large image dataset. Profiling shows the GPU sits idle for several hundred milliseconds between batches. The DataLoader currently uses num_workers=0. Which configuration change will most directly reduce the inter-batch GPU idle time?
- AIncrease the batch size substantially so each GPU kernel runs longer, amortising the fixed per-batch data transfer overhead across more samples per step.
- BReplace the default collate function with a custom one that stacks tensors directly on the GPU, removing the explicit host-to-device copy step from inside the training loop.
- CMove the entire dataset into GPU VRAM at the start of training using a pre-loaded TensorDataset so that each batch requires no host-to-device transfer during the loop.
- DSet num_workers to a value greater than zero and enable pin_memory=True so that background worker processes prefetch batches while the GPU trains, and pinned memory enables faster asynchronous DMA transfers. Correct
Why A is wrong: A larger batch size reduces the number of host-to-device transfers per epoch but does not introduce prefetching. The root cause is that data loading is sequential and blocking, and increasing batch size does not change that. It may also cause out-of-memory errors or destabilise training.
Why B is wrong: DataLoader workers run in forked processes that cannot share CUDA contexts in the standard configuration; attempting to create GPU tensors inside a worker raises a RuntimeError. Tensors must be transferred to the GPU inside the main training loop, not during collation in the worker process.
Why C is wrong: Pre-loading a large image dataset into GPU VRAM is not feasible; VRAM capacity is far smaller than typical dataset sizes and the approach fails with an out-of-memory error. It also prevents standard CPU-side augmentation pipelines from running, limiting training data diversity.
Why D is correct: With num_workers=0, the main process blocks on each batch load before resuming the training loop, serialising CPU work and GPU computation. Setting num_workers greater than zero spawns workers that prepare the next batch in parallel. pin_memory=True allocates output tensors in page-locked host memory, enabling the CUDA DMA engine to transfer them asynchronously with higher bandwidth, further overlapping data preparation with GPU execution.