Data-Engineer-Associate - Troubleshooting, Monitoring, and Optimization (10% of the exam) - Section 6.5

Diagnose cluster startup failures, library conflicts and out-of-memory issues.

Diagnose why compute fails to start (quotas, instance availability, init scripts, policies), why a library install conflicts with the runtime, and why a driver or executor runs out of memory, then pick the fix that addresses the cause, such as avoiding collect() on large data.

compute startup failuresinit scriptslibrary conflictsdriver out of memorycompute policies

Practice question for this objective

Free sampleTroubleshooting, Monitoring, and Optimizationhard

A nightly Lakeflow Job stopped producing any task output last week. Every run ends within a few minutes and the workspace reports that the job cluster was terminated because a cluster-scoped init script failed. The script was added seven days ago and installs an operating system package from an internal package repository. The engineer on call must establish why the script failed on this compute before changing the script or the cluster. Which action supplies that evidence?

AAttach a notebook to the job cluster and step through the init script commands one at a time so that the failing command reveals itself interactively.
BOpen the Spark UI for the failed run and read the executor and stage detail pages, where the error raised by the init script is recorded against the first stage.
CIncrease the driver node size and the autoscaling maximum on the job cluster, then restart the run to see whether the script completes with more resources available to it.
DRead the termination reason recorded in the cluster event log for the failed cluster, then open the init script output written to the destination configured for cluster log delivery. Correct

Diagnose a cluster that fails to start on an init script by reading the cluster event log and the delivered init script logs. Init scripts run on each node during cluster startup, before the Spark application exists. A non-zero exit from one terminates the cluster, so the only record of what happened is the cluster event log entry naming the failure plus the script's own standard output and standard error, which are captured only where cluster log delivery has been configured.

Why A is wrong: This is how the script would be developed in the first place, which makes it tempting, but the cluster never reaches a running state when an init script fails, so no notebook can attach to it and no cell can execute.

Why B is wrong: The Spark UI is the right place for a failure inside a running application, but init scripts execute before the Spark application starts, so there is no application, no stage and no executor page for this run at all.

Why C is wrong: Resizing is a plausible reflex when a cluster will not start, but it is a cause-blind fix: a script that fails on a package download or a permission error fails identically on larger instances, and the evidence is still unread.

Why D is correct: The event log names the failure and identifies the script, and the delivered init script log holds the standard output and standard error of the script itself, which is where the failing command and its message appear.

See more Data-Engineer-Associate practice questions, answers explained.

Exam traps in Troubleshooting, Monitoring, and Optimization

Answers that look right on this material and are not. Each one is a distractor from a different question in the Data-Engineer-Associate bank for this domain.

Executor memory is too small to hold the broadcast copy of the dimension side, so the shortfall is reported by the driver; raise spark.executor.memory until each executor can hold the whole dimension.
Why it is wrong: Executors do hold a copy of a broadcast relation, which makes this plausible, but the stem reports low executor memory pressure and an error raised in the driver, so raising executor memory addresses a component that is not under strain.
The newer version replaces the cluster library for every notebook attached to the cluster, so the analysts pick it up as soon as they execute their next cell against the shared compute.
Why it is wrong: This confuses a notebook-scoped install with a cluster library. Cluster libraries are installed through the compute configuration and apply to all sessions, whereas a %pip install acts on the calling notebook's own environment and leaves the others untouched.
The script failure is recorded as a warning, the node carries on starting, and the engineer should read the notebook cell output for the failed package download.
Why it is wrong: This is tempting because a package install can look like an optional extra rather than a gate on the node coming up. It is wrong because the init script runs during node startup, well before any notebook is attached, and its exit status is treated as a hard result rather than advisory.

Examworthy is not affiliated with or endorsed by Databricks. Original, blueprint-aligned practice material only.