Optimise resources to minimise cost while meeting business needs, managing resource availability for critical processes and deciding between persistent and job-based clusters - PDE Maintaining and Automating Data Workloads

Free sampleMaintaining and Automating Data Workloadsmedium

A platform team is choosing between a persistent Dataproc cluster with autoscaling and Dataproc ephemeral job clusters for a portfolio of Spark batch jobs that vary in size, run on independent schedules, and share no in-cluster state. Cost per job and isolation between tenants are the primary criteria. Which statement best captures the trade-off they should weigh?

AA persistent cluster with autoscaling always minimises cost per job because it amortises master node cost across many jobs, and isolation is automatic because YARN queues separate tenant resources.
BEphemeral job clusters always cost more than a persistent cluster because of repeated start-up overhead, so persistent clusters are preferred whenever workloads run more than once a week regardless of tenant isolation needs.
CEphemeral job clusters give strong tenant isolation and you only pay for the runtime of each job, but they incur cluster start-up latency and miss the benefit of warm caches and reused initialisation work between jobs. Correct
DPersistent clusters and ephemeral job clusters have identical cost characteristics once autoscaling is enabled on the persistent cluster, so the choice is purely a matter of operational preference.

Compare persistent autoscaled Dataproc clusters with ephemeral job clusters on cost per job, isolation, and start-up overhead. Dataproc job clusters created per job stop billing when the job ends and give each job its own master, workers, image, and libraries, which is strong tenant isolation. The cost is start-up latency on every run and the loss of warm caches and reused state. A persistent autoscaled cluster avoids start-up latency and can amortise initialisation, but it pays for the master and minimum workers between jobs and shares the runtime environment across tenants.

Why A is wrong: Tempting because amortised master cost sounds attractive, but a persistent cluster keeps the master and minimum workers running between jobs, and YARN queues do not provide the same isolation as separate clusters when jobs have different library or image requirements.

Why B is wrong: Tempting because start-up time is real, but start-up cost is small compared with paying for an idle persistent cluster between sparse jobs, and the absolute rule ignores the isolation criterion the team explicitly cares about.

Why C is correct: Correct. A new cluster per job isolates dependencies and noisy neighbours and stops the meter when the job ends, but it pays start-up latency every run and cannot reuse cached data, shuffle output, or warm JVMs, which a persistent cluster can.

Why D is wrong: Tempting because autoscaling does narrow the gap, but it does not eliminate it. A persistent cluster still keeps the master and the minimum worker count running between jobs, and isolation properties differ materially.

Optimise resources to minimise cost while meeting business needs, managing resource availability for critical processes and deciding between persistent and job-based clusters.

Practice question for this objective

More in this domain