NCA-ADS - Data Science Pipelines and Workflow Automation (13% of the exam) - Section 3.4
Automate and scale workflows with RAPIDS and Dask.
Scale data science workflows beyond a single GPU by combining RAPIDS with Dask to distribute computation across multiple workers. Configure Dask clusters, partition data appropriately, and recognise when task-graph overhead outweighs the parallelism benefit.
RAPIDSDask
Practice question for this objective
lock_openFree sampleData Science Pipelines and Workflow Automationhard
A team has a cuDF-based ETL pipeline that runs correctly on a single GPU but must now handle datasets that exceed available VRAM. They refactor the code to use dask_cudf. During testing they find that increasing the number of partitions beyond what the data volume strictly requires causes a measurable rise in total execution time even though each partition now fits comfortably within VRAM. Which explanation best accounts for this overhead?
AEach additional partition introduces task-graph nodes that the Dask scheduler must serialise, communicate, and track, adding scheduling and inter-worker coordination overhead that accumulates with partition countcheck_circle Correct
BMore partitions increase the total GPU memory reserved per worker because dask_cudf pre-allocates a fixed buffer for each partition regardless of its actual size
Cdask_cudf converts each partition to a pandas DataFrame before sending it to a worker, and more partitions multiply the host-to-device transfer cost
DIncreasing partition count forces the Dask scheduler to serialise all partitions through a single worker to maintain partition ordering, eliminating parallelism
Explain how partition count affects Dask task-graph scheduling overhead and the trade-off when sizing partitions for dask_cudf workloads. In a Dask workload each partition generates one or more nodes in the task graph. The scheduler incurs overhead for every node: it must serialise the graph, dispatch tasks to workers, handle result metadata, and coordinate any shuffle or aggregation steps. When partitions are smaller than necessary the workload generates far more graph nodes than the computation warrants, and the cumulative scheduling cost exceeds the parallelism gained. The practical guidance for dask_cudf is to target partition sizes that keep each GPU worker comfortably occupied - typically tens to hundreds of megabytes per partition - so that task-graph overhead remains a small fraction of actual compute time.
Why A is correct: Every partition maps to at least one node in the Dask task graph. The scheduler must plan, dispatch, and track each node; inter-worker data movement and result aggregation also scale with partition count. When partitions are too small the coordination overhead outweighs the parallelism benefit, making an over-partitioned graph slower than a well-sized one.
Why B is wrong: dask_cudf does not pre-allocate fixed-size buffers per partition; memory is allocated when a partition is actually loaded, so partition count alone does not inflate reserved memory in the way described.
Why C is wrong: dask_cudf keeps data as cuDF DataFrames on the GPU workers and does not convert partitions to pandas as a standard step; claiming a pandas conversion occurs conflates the dask_cudf path with the CPU-based Dask DataFrame path.
Why D is wrong: The Dask scheduler does not route all partitions through a single worker to maintain order; partitions are processed concurrently across workers and ordering constraints are handled at the graph level only when a shuffle or sort is explicitly requested.
Exam traps in Data Science Pipelines and Workflow Automation
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-ADS bank for this domain.
Assigning the dask_cudf DataFrame to a Python variable
errorWhy it is wrong: Assignment is a Python name-binding operation and has no effect on Dask scheduling; the task graph remains unevaluated and no GPU work is dispatched at that point.
Start a plain Dask LocalCluster with four worker processes, since each generic worker will detect and claim one of the four GPUs automatically when GPU tasks arrive
errorWhy it is wrong: A generic LocalCluster spawns CPU-oriented workers with no device binding; they will not each claim a distinct GPU automatically, so GPU work is not reliably spread one-per-device and GPU-aware transfers are absent.
Switching the cuDF reader to load the Parquet file in smaller row groups so each partition is guaranteed to stay smaller than the GPU memory throughout
errorWhy it is wrong: Smaller row groups reduce typical partition size, but they do not address transient peaks during the group-by where intermediate buffers spike, so a few partitions can still overflow device memory and abort the job.