Data-Engineer-Associate - Troubleshooting, Monitoring, and Optimization (10% of the exam) - Section 6.3

Identify common performance bottlenecks such as data skew, shuffling and disk spilling by interpreting stage-level metrics in the Spark UI.

Read stage and task metrics in the Spark UI, including the task summary's min, median and max, shuffle read and write, and spill, to diagnose skew, excessive shuffling and memory pressure. Choose the fix that addresses the cause, such as adaptive skew join handling, rather than adding hardware.

Spark UIdata skewshuffle readdisk spilladaptive skew join

Practice question for this objective

Free sampleTroubleshooting, Monitoring, and Optimizationhard

A daily aggregation over a Delta table runs on a cluster providing 32 cores in total. In the Spark UI the stage that reads the shuffle reports 8,000 tasks, a median task shuffle read of 3 MB, a maximum of 4 MB, no spill recorded, and a median task duration under one second, while the stage itself takes fourteen minutes from start to finish. The cluster inherits a Spark configuration from an older workload, in which spark.sql.shuffle.partitions is set to 8000 and spark.sql.adaptive.enabled is set to false. Which change addresses the cause of that stage duration?

spark.sql.shuffle.partitions 8000
spark.sql.adaptive.enabled false

ARaise spark.sql.shuffle.partitions to 16000, so that the stage divides its work across more tasks and makes fuller use of the cores already available on the cluster.
BSet spark.sql.adaptive.enabled to true, so that adaptive partition coalescing merges the small post shuffle partitions into fewer and larger tasks while the query is running. Correct
CSet spark.sql.adaptive.skewJoin.enabled to true, so that the oversized partitions in the stage are split into smaller ones and spread over cores that are otherwise idle.
DDouble the cluster to 64 cores, so that twice as many of the 8,000 tasks are in flight at once and the stage finishes in roughly half of its present duration.

Recognise that a stage of very many tiny even tasks is per task overhead, cured by adaptive partition coalescing rather than by more partitions or more hardware. A partition count fixed at 8000 against roughly 24 GB of shuffle data yields 3 MB partitions, and each task carries fixed costs for scheduling, launching, fetching its shuffle blocks and reporting results. With those costs dominating a sub second task, the stage spends its fourteen minutes on overhead rather than on computation. Adaptive query execution measures the actual shuffle output at run time and coalesces the small partitions into a smaller number of larger tasks, which removes most of that overhead without any change to the configured partition count.

Why A is wrong: It is tempting because more partitions usually means more parallelism, but the partition count already exceeds the core count by a factor of 250, so doubling it doubles the scheduling and shuffle block bookkeeping that is producing the fourteen minutes.

Why B is correct: Adaptive query execution reads the shuffle statistics after the map side finishes, and with spark.sql.adaptive.coalescePartitions.enabled at its default it combines the tiny partitions into a task count suited to the data volume and the cluster.

Why C is wrong: Skew handling is a reasonable reflex when a stage runs long, but the reported median of 3 MB and maximum of 4 MB show an even distribution with no oversized partition to split, and that setting has no effect while adaptive query execution is switched off.

Why D is wrong: Adding hardware looks attractive because the tasks queue behind the available cores, but it treats the symptom and leaves the per task overhead untouched, so the run costs twice as much for a much smaller improvement than the arithmetic suggests.

See more Data-Engineer-Associate practice questions, answers explained.

Exam traps in Troubleshooting, Monitoring, and Optimization

Answers that look right on this material and are not. Each one is a distractor from a different question in the Data-Engineer-Associate bank for this domain.

It broadcasts the skewed side of the join to every executor, so the oversized partition is read locally instead of being shuffled across the network.
Why it is wrong: Tempting because broadcasting is the other adaptive join change, but it applies when one whole side is small enough for the broadcast threshold. A partition of 22 GB is the opposite case.
Leave the query as it stands and rely on the adaptive skew join optimisation to divide the oversized partition, since adaptive query execution is already active on this cluster.
Why it is wrong: Tempting because adaptive query execution does split oversized shuffle partitions, but its skew handling applies to sort merge join stages, and the heavy stage here is a grouped aggregation rather than a join, so nothing splits the guest checkout partition.
They count different records: the memory figure covers rows evicted from the executor storage cache, while the disk figure counts the shuffle blocks this stage wrote to local disk for the next stage to fetch.
Why it is wrong: Tempting because cached data really can be evicted and shuffle blocks really are written to local disk, but neither is what spill records. Spill is execution memory being flushed mid stage, and the bytes written for the next stage are reported separately as shuffle write.

Examworthy is not affiliated with or endorsed by Databricks. Original, blueprint-aligned practice material only.