Data-Engineer-Associate domain - 10% of the exam

Troubleshooting, Monitoring, and Optimization

Troubleshooting, Monitoring, and Optimization is 10% of the Data-Engineer-Associate exam. These are the objectives it covers, each with practice questions, with every answer explained.

The domain in numbers

30
Practice questions
5
Objectives
17
Medium
13
Hard

Where people go wrong

  • Using a timeout or retry where a duration warning is needed

    The wrong answers meet a slow run with a control that cancels it, retries it or waits for it to fail. A timeout cancels the run when it is reached, retries fire only after a task errors, and a failure notification never fires for a run that succeeds slowly. A duration warning threshold on the job sends a notification while the run is still executing and lets it carry on, which is the setting for being told early without losing the output.

    See it in a sample question
  • Blaming task execution for time spent before the first task starts

    The wrong answers put a growing run duration down to spill, skew, growing input, retries or a lowered worker maximum, and each of those would lengthen a task's own reported duration. Total run duration also covers queueing and compute provisioning. When per-task durations stay flat and the gap before the first task grows, the time is going on acquiring compute, so cluster start and queueing are the first thing to address.

    See it in a sample question
  • Searching a skipped task for the fault

    The wrong answers treat a task marked skipped, or one reporting an upstream dependency failure, as a task that ran and errored, with logs, partial writes or a timeout of its own. A skipped task was never dispatched, because status passes from a failed dependency down to its dependants and never upstream. Only the failed task ran, so its own logs and error output hold the cause, and the investigation starts there.

    See it in a sample question
  • Adding partitions or workers for one oversized key

    The wrong answers raise the shuffle partition count or add workers when one task in a stage is far larger than the rest. Every row sharing one customer_id or account_id hashes to a single partition whatever the partition count, and one partition is processed by one task, so extra workers sit idle. Salting the heavy key spreads it across many tasks, and enabling adaptive query execution splits an oversized join partition at run time.

    See it in a sample question
  • Treating liquid clustering as an addition to partitioning

    The wrong answers layer liquid clustering on top of partitioning or Z-ordering, or expect a changed clustering key to rewrite existing files at once. Liquid clustering replaces both: a clustered table cannot declare partition columns and ZORDER BY is unsupported on it. It creates no directory per key value. ALTER TABLE with a new CLUSTER BY changes the keys for later writes, while files already written keep their layout until a later OPTIMIZE run.

    See it in a sample question

Objectives in this domain

Sample questions from this domain

Free sampleTroubleshooting, Monitoring, and Optimizationmedium

A Lakeflow Job normally finishes in about twenty five minutes and feeds a report published at 07:00. Reviewing the run history, a team finds several runs in recent weeks that took over an hour but still succeeded, and nobody was told until the report was late. The team wants to be notified while a run is still going once it passes forty minutes, and it does not want a slow run cancelled, because a late report is preferable to no report. Which configuration meets the requirement?

  • ASet a timeout of forty minutes on the job and send a notification when it is exceeded, so the team learns about the slow run as soon as the threshold is passed.
  • BSet a duration warning threshold of forty minutes on the job and send a notification when it is exceeded, leaving the run to continue to completion. Correct
  • CAdd a notification on run failure to the job, so that any run departing from its usual duration is reported to the team while it is still executing.
  • DAdd a retry policy of two attempts to the job, so that a slow attempt is abandoned and a fresh attempt of the usual duration produces the report on time.
Use a job duration warning threshold to be alerted about a slow but successful run without cancelling it, unlike a timeout. Lakeflow Jobs lets a job or task carry a duration threshold, and the warning form and the timeout form behave differently: the warning fires a notification when the run passes the configured duration and lets the run carry on, while the timeout treats the threshold as a hard limit and cancels the run. Because these runs succeed, failure notifications never fire, and because the problem is duration rather than a transient error, retries are the wrong control. Only the warning threshold surfaces the emerging trend early while still producing the report.

Why A is wrong: A timeout does notify at the threshold, which makes it look like the same control, but it also cancels the run, so the report the team wanted late would never be produced.

Why B is correct: A warning threshold raises a notification while the run continues, which is exactly the split the team asked for between being told early and letting the slow run finish.

Why C is wrong: Failure notifications are already common practice and cost nothing to add, but these runs succeed, so the condition never fires and the long runs stay invisible.

Why D is wrong: Retries do help a run that fails on a transient fault, but they are not triggered by duration, and abandoning work in progress would push the report later rather than earlier.

Free sampleTroubleshooting, Monitoring, and Optimizationmedium

An hourly Lakeflow Job has seen its end to end run duration rise from about twelve minutes to about thirty over a fortnight, with every run still succeeding. In the job's run history the duration reported for each individual task is unchanged, and the run timeline shows a long stretch at the beginning of every recent run before the first task starts executing. Which conclusion does that evidence support?

  • AThe additional time is spent acquiring compute before any task is dispatched, so the compute the job attaches to is where to look rather than the code in the tasks. Correct
  • BThe first task in the graph has developed data skew, and one long running task at the head of the graph is holding back everything that depends on it.
  • CThe failure rate of the job has risen, and the retry attempts made against the failing tasks account for the additional time in each recent run.
  • DThe source tables have grown over the fortnight, and the extra scan time is recorded outside the individual task durations reported in the run history.
A job run's duration includes waiting for compute, so unchanged task durations point at provisioning rather than at code. The duration recorded for a job run spans the whole run, including the period spent starting or waiting for the compute the tasks attach to, whereas a task's own duration covers only its execution. When task durations hold steady while the run grows and the timeline shows a widening gap before the first task begins, the added time sits in compute acquisition, so the cluster configuration or availability is the thing to investigate.

Why A is correct: Correct: a run's duration covers the wait for compute as well as task execution, so unchanged task durations plus a growing gap before the first task points at provisioning or queueing.

Why B is wrong: Tempting because skew is a common cause of a stage dragging, but skew would show as a longer duration for the task itself, and every task duration here is unchanged.

Why C is wrong: Tempting because retries do lengthen a run, but the runs are all succeeding and a retry would show inside the affected task's own reported duration rather than before the first task.

Why D is wrong: Tempting because the sources probably are growing, but scan work is performed by a task and is counted in that task's reported duration, so growth would move those figures.

Free sampleTroubleshooting, Monitoring, and Optimizationmedium

A Lakeflow Job holds an ingest task, a transform task that depends on ingest, and a publish task that depends on transform. Overnight the run finished and the job run is marked as failed. In the Lakeflow Jobs UI the run's task graph shows ingest with a failed status, while transform and publish each carry a status reporting that an upstream dependency failed. An on-call engineer has to decide where to begin. What does that pattern indicate?

  • AThe transform and publish tasks each ran and raised their own errors after reading partial output written by ingest, so the three task logs have to be read together before a cause emerges.
  • BThe transform and publish tasks were never started, so the run has a single point of failure and the ingest task's own error output is where the investigation belongs. Correct
  • CThe transform and publish tasks were cut short when the job cluster lost workers partway through the run, so the compute event log for that cluster is the place to look for the cause.
  • DThe transform and publish tasks exceeded their own time limits while queued behind ingest on the shared job cluster, so raising the timeout configured on each of them clears the problem.
A downstream task reporting an upstream dependency failure never ran, so the investigation belongs in the failed upstream task. Lakeflow Jobs evaluates task dependencies before launching a task. When a dependency fails, the dependent tasks are not launched and are marked as blocked by that upstream failure, so they emit no logs of their own and hold no partial results. The whole run therefore traces back to the single task that genuinely failed, which is why the task graph is read from the failure point downwards rather than task by task.

Why A is wrong: Tempting because a partial write really can poison downstream logic, but the upstream failure status is reported precisely when a task is not launched, so neither downstream task produced an error of its own.

Why B is correct: The task graph propagates a failure downward, so a dependency that fails blocks every task hanging off it and leaves those tasks unexecuted, which narrows the investigation to the one task that actually failed.

Why C is wrong: A worker loss is a real cause of mid run failures, but it would surface as a failed or cancelled task with its own driver output, rather than as a dependency status attached to two unstarted tasks.

Why D is wrong: Waiting on a dependency is not counted against a task's timeout, and a timed out task carries its own timeout status, so raising these limits changes nothing about a run blocked by a failed upstream task.

Free sampleTroubleshooting, Monitoring, and Optimizationhard

A daily job joins a 1.8 TB fact table to a 400 GB dimension table on account_id. In the Spark UI the sort merge join stage reports a median task shuffle read of 160 MB and a maximum of 19 GB, and one task is still running an hour after the other 199 have finished. The cluster carries a Spark configuration copied from an older cluster, in which spark.sql.adaptive.enabled is set to false. An engineer proposes rewriting the join around a salted account_id. What should the engineer do first, and why?

spark.sql.adaptive.enabled false
spark.sql.shuffle.partitions 200
  • ASet spark.sql.adaptive.enabled to true and rerun the job, because adaptive query execution divides an oversized shuffle partition into several tasks at run time and may remove the need to alter the query. Correct
  • BApply the salted account_id straight away, because a maximum task shuffle read of 19 GB against a median of 160 MB shows the key distribution itself has to be rewritten in the query.
  • CRaise spark.sql.autoBroadcastJoinThreshold above the size of the dimension table, so that the join runs as a broadcast hash join and the shuffle of the fact side disappears.
  • DRaise spark.sql.shuffle.partitions from 200 to 2000, so that the join shuffle produces smaller partitions and the heavy account_id is spread over a much larger number of them.
Confirm that adaptive query execution and its skew join handling are switched on before rewriting a skewed join by hand. Adaptive query execution reads shuffle statistics after the map side has run, and its skew join handling splits a partition that is far larger than the median into several tasks that each join against a replicated copy of the matching build side rows. A cluster configuration that sets spark.sql.adaptive.enabled to false disables that behaviour, so the classic single long task reappears. Checking the setting costs one rerun, whereas salting changes the query permanently.

Why A is correct: Correct, because the cluster has switched off the run time optimisation designed for exactly this pattern, and restoring it lets the skewed join partition be split across tasks without a rewrite that the team would then have to maintain.

Why B is wrong: Tempting because the metrics do confirm genuine skew, but salting is the manual fallback for cases the platform cannot handle, and reaching for it while the built in skew handling is disabled buys complexity that a configuration change would have avoided.

Why C is wrong: Tempting because broadcasting a dimension does remove the shuffle, but a 400 GB dimension cannot be collected to the driver and sent to every executor, so the change would replace a slow task with a failing job.

Why D is wrong: Tempting because the stage clearly holds one partition that is far too large, but a single account_id hashes to one partition at any partition count, so the long task survives and the other tasks become inefficiently small.

Free sampleTroubleshooting, Monitoring, and Optimizationmedium

A Delta table created with CLUSTER BY on order_date has been loaded daily for several months. Reporting has shifted to filtering on customer_id, so an engineer runs the statement below and then loads the next daily batch into the table. What is the state of the table's data immediately after that batch has been written?

ALTER TABLE prod.sales.orders CLUSTER BY (customer_id);
  • AEvery file already in the table is rewritten on customer_id as part of the ALTER TABLE statement, so the whole table is clustered on the new key before the daily batch is loaded into it at all.
  • BThe statement alters table metadata alone and influences no write at all, so nothing in the table is clustered on customer_id until the engineer additionally runs OPTIMIZE with ZORDER BY on the customer_id column.
  • CThe table now carries order_date and customer_id together as its clustering keys, because ALTER TABLE with CLUSTER BY adds the named columns to the existing key set rather than replacing what was declared before.
  • DThe rows in the new batch are clustered on customer_id as they are written, while the data written before the change keeps the layout it already had until a later maintenance run reorganises it on the new key. Correct
Changing liquid clustering keys applies to new writes at once and leaves older data to be reorganised by later maintenance runs. Clustering keys are held as table metadata, so ALTER TABLE replaces the recorded key list without touching stored files. Writes that follow the change organise their records on the new key, and the files written under the old key are reclustered gradually by later OPTIMIZE work, which is what makes revising the keys a cheap operation on a large table.

Why A is wrong: A candidate may expect a layout change to be applied eagerly, but ALTER TABLE only updates the clustering key metadata; rewriting months of files inside a metadata statement would make the operation arbitrarily expensive.

Why B is wrong: This is tempting because the statement is indeed a metadata change, but the recorded key governs later writes, and Z-ordering is a separate technique that is not used to service a liquid clustering key.

Why C is wrong: Additive behaviour is a reasonable guess given that several clustering keys are permitted, but the clause states the complete key list, so the previous key is replaced unless it is named again in the statement.

Why D is correct: Correct: the new key takes effect for incoming writes straight away, and the older files are reorganised incrementally by subsequent OPTIMIZE work rather than by the ALTER TABLE statement itself.

Other domains in this exam

See also the Data-Engineer-Associate cert hub, the study guide, and the cheat sheet.

Examworthy is not affiliated with or endorsed by Databricks. Original, blueprint-aligned practice material only.