A team has a cuDF-based ETL pipeline that runs correctly on a single GPU but must now handle datasets that exceed available VRAM. They refactor the code to use dask_cudf. During testing they find that increasing the number of partitions beyond what the data volume strictly requires causes a measurable rise in total execution time even though each partition now fits comfortably within VRAM. Which explanation best accounts for this overhead?
- AEach additional partition introduces task-graph nodes that the Dask scheduler must serialise, communicate, and track, adding scheduling and inter-worker coordination overhead that accumulates with partition count Correct
- BMore partitions increase the total GPU memory reserved per worker because dask_cudf pre-allocates a fixed buffer for each partition regardless of its actual size
- Cdask_cudf converts each partition to a pandas DataFrame before sending it to a worker, and more partitions multiply the host-to-device transfer cost
- DIncreasing partition count forces the Dask scheduler to serialise all partitions through a single worker to maintain partition ordering, eliminating parallelism
Why A is correct: Every partition maps to at least one node in the Dask task graph. The scheduler must plan, dispatch, and track each node; inter-worker data movement and result aggregation also scale with partition count. When partitions are too small the coordination overhead outweighs the parallelism benefit, making an over-partitioned graph slower than a well-sized one.
Why B is wrong: dask_cudf does not pre-allocate fixed-size buffers per partition; memory is allocated when a partition is actually loaded, so partition count alone does not inflate reserved memory in the way described.
Why C is wrong: dask_cudf keeps data as cuDF DataFrames on the GPU workers and does not convert partitions to pandas as a standard step; claiming a pandas conversion occurs conflates the dask_cudf path with the CPU-based Dask DataFrame path.
Why D is wrong: The Dask scheduler does not route all partitions through a single worker to maintain order; partitions are processed concurrently across workers and ordering constraints are handled at the graph level only when a shuffle or sort is explicitly requested.