A nightly Lakeflow Job has grown from about forty minutes to about seventy minutes over the past three weeks. Opening the job in the Lakeflow Jobs run history, an engineer sees that the transformation task itself still takes roughly the same time in every recent run, while the gap between the run starting and the task beginning has grown steadily. The job runs on a job cluster that is created for each run. Which conclusion does the run history support, and what should the engineer address first?
- AThe transformation is spilling more shuffle data each night, so the engineer should raise the executor memory on the job cluster and rerun to confirm the improvement.
- BThe source tables have grown, so the engineer should repartition the largest input and add workers to the job cluster until the nightly duration returns to its earlier level.
- CThe job is being throttled by concurrent runs of the same job, so the engineer should lower the maximum concurrent runs setting and rerun the job on the same schedule.
- DThe extra time is spent before execution, waiting on compute to be provisioned, so the engineer should attack the cluster start and queueing time rather than tune the Spark work. Correct
Why A is wrong: Spill is a genuine cause of growing task duration and is a common first guess, but the run history shows the task duration flat and the pre-task time rising, so the extra minutes are not being spent inside the transformation at all.
Why B is wrong: Growing input would show up as longer task execution in the run history, and it does not, so adding workers spends money on the part of the run that has not changed.
Why C is wrong: Concurrency limits can delay a run and so this looks plausible, but a single nightly schedule produces one run at a time, and the run history would show a queued run rather than a steadily lengthening setup phase.
Why D is correct: The run history separates the waiting and setup portion of a run from the task execution portion, and here only the former has grown, which points at compute acquisition rather than at the transformation logic.