Data-Engineer-Associate domain - 16% of the exam

Working with Lakeflow Jobs

Working with Lakeflow Jobs is 16% of the Data-Engineer-Associate exam. These are the objectives it covers, each with practice questions, with every answer explained.

The domain in numbers

48
Practice questions
4
Objectives
24
Medium
24
Hard

Where people go wrong

  • Expecting a task to run after a failed dependency

    The wrong answers reach for retries, a condition task, At least one succeeded or None failed to make a final task run whatever happened upstream. The default Run if condition, All succeeded, skips a task when any dependency fails or is skipped, and a condition task placed downstream of the failure is skipped too. All done starts the task once every dependency has reached a terminal state, which suits audit, cleanup and notification tasks.

    See it in a sample question
  • Rerunning the whole job instead of repairing the run

    The wrong answers start a new run, edit the job definition or add retries after the failure, and each either repeats tasks that already succeeded or has no effect on a run that has ended. Run now executes the full graph from the start, and retries apply while a run is in progress. A repair run reuses the successful tasks, re-executes the failed task and its dependants, and records the outcome against the original run.

    See it in a sample question
  • Using a file arrival trigger for a table dependency

    When the load of a silver Unity Catalog table finishes at drifting times, the wrong answers offer a file arrival trigger, continuous execution, a padded cron time or a frequent schedule that checks for a new version. The file arrival trigger is meant for a landing location of raw files and reacts to file activity rather than to a committed table version. Continuous execution spends compute while the table is unchanged. A table update trigger starts a run when the named table records a new commit.

    See it in a sample question
  • Treating the file arrival trigger as a file handler

    The wrong answers expect the trigger to pass the task a list of files or to start one run for each file. The trigger only starts a run when the monitored location holds files it has not seen, so files that gathered during a pause produce a single run. Processing each file once stays with the task, through Auto Loader checkpoints or the COPY INTO load history.

    See it in a sample question
  • Treating task values as general storage

    The wrong answers make a task value depend on shared compute, carry over into the next run, or hold bulk data. The jobs service keeps a task value for the duration of one run, so separate clusters can read it and a fresh run starts with none. Values suit small control data, so a list of 30,000 identifiers belongs in a Unity Catalog table with only a batch identifier published as the value.

    See it in a sample question

Objectives in this domain

Sample questions from this domain

Free sampleWorking with Lakeflow Jobshard

A Lakeflow Job runs three ingestion tasks in parallel, and a final task named notify depends on all three. The team requires notify to post a run summary on every run, including runs where one ingestion task fails and runs where a task is skipped, and it must not be held back when a dependency ends in a state other than success. Which Run if condition on notify meets the requirement?

  • AAt least one succeeded, which starts notify as soon as any dependency has succeeded and leaves it skipped on a run in which every ingestion task fails.
  • BNone failed, which starts notify when every dependency has finished in a state other than failure and leaves it skipped as soon as one ingestion task fails.
  • CAll succeeded, the default, which starts notify only when all three ingestion tasks have succeeded and leaves it skipped whenever one fails or is skipped.
  • DAll done, which starts notify once every dependency has reached a terminal state, whether that state is success, failure or skipped. Correct
Use the All done Run if condition for a notification or cleanup task that must run whatever the outcome of its dependencies. The Run if condition governs whether a dependent task starts, based on the terminal states of its dependencies. All done is the only setting that ignores those states entirely and waits purely for completion, which is what a task that must report on failures as well as successes needs.

Why A is wrong: It loosens the default enough to survive a single failure, but it still needs one success, so the run where all three ingestion tasks fail is exactly the run that produces no summary.

Why B is wrong: This tolerates skipped dependencies, which is half of the requirement, but a single failed ingestion task suppresses the summary, and failures are the case the team most wants reported.

Why C is wrong: This is the setting already in place and the reason no summary appears on a bad night, since any non-successful dependency skips the dependent task.

Why D is correct: All done waits only for the dependencies to finish and ignores the outcome, which is the condition designed for cleanup and notification tasks that must run on every path.

Free sampleWorking with Lakeflow Jobshard

A Lakeflow Job of eight dependent tasks fails at the sixth task because of a defect in one notebook. The engineer corrects the notebook. The five tasks that already succeeded took over an hour and wrote to Unity Catalog tables, so re-executing them would waste compute and rewrite data that is already correct. The operations team also wants the outcome recorded against the original run rather than appearing as a separate one. What should the engineer do?

  • AStart a repair run from the failed run, which re-executes the failed task and the tasks that depend on it while leaving the already successful tasks untouched, all recorded under the same job run. Correct
  • BTrigger the job again with Run now, since a fresh run detects the tables that the first five tasks already populated and skips those tasks automatically before continuing from the sixth.
  • CRaise the maximum retry count on the sixth task and wait for the run to retry it, since retries resume the failed run from the point of failure once the notebook has been corrected.
  • DRemove the five completed tasks from the job definition, run the shortened job, and add the removed tasks back afterwards so that only the outstanding work is executed this time.
Use a repair run to re-execute only the failed tasks and their dependants within an existing job run after fixing the underlying defect. A repair run operates on a completed job run rather than starting a new one. It re-executes the tasks that failed, plus the downstream tasks that were skipped because of them, and treats the successful tasks as already done, so their expensive output is reused. Because the repair is attached to the original run, the run history holds one entry showing the eventual outcome.

Why A is correct: Correct. A repair run is designed for precisely this case: it resumes an existing run from its failed tasks, reuses the successful results, and keeps one run record for the operations team.

Why B is wrong: Run now creates a new run that executes the whole graph from the start. Lakeflow Jobs does not inspect target tables to decide which tasks to skip, so the first five tasks would run again.

Why C is wrong: Retries are consumed while the run is still active and repeat the same code, so a run that has already terminated as failed cannot be resumed this way and a code fix is not picked up retrospectively.

Why D is wrong: Editing the definition would execute the outstanding work, but it changes a shared production job for everyone, loses the dependency structure temporarily, and still produces a separate run record.

Free sampleWorking with Lakeflow Jobsmedium

A gold aggregation job must run after a silver Unity Catalog table has been loaded by a separate pipeline whose finishing time drifts between 01:10 and 03:40 depending on volume. The team will not guess a safe cron time and will not merge the two workloads into a single job. Which trigger fits the gold job, and what does that trigger act on?

  • AA file arrival trigger pointed at the storage path behind the silver table, which starts a run whenever a fresh data file is written beneath that path by the loading pipeline.
  • BA continuous trigger, which keeps one run of the gold job active at all times so the published aggregate is never more than a moment behind whatever the silver load has committed.
  • CA scheduled trigger set to 03:45, which is the sound choice here because a table update trigger responds to schema changes on the watched table rather than to newly committed data.
  • DA table update trigger, which starts a run once the named Unity Catalog table records a new commit, so the gold job follows the silver load whenever that load happens to finish. Correct
A table update trigger starts a Lakeflow Job when a watched Unity Catalog table receives a new commit, coupling downstream work to upstream completion. A table update trigger monitors named Unity Catalog tables and raises a job run when one of them records a new commit. That makes the downstream job react to the completion of the upstream write itself, so a variable finishing time upstream needs no padded cron guess and no merging of the two workloads into a single job graph.

Why A is wrong: Plausible because a Delta write does produce files, but the file arrival trigger is meant for a registered external location or volume holding incoming data, and reacting to table commits is what the table update trigger is designed for.

Why B is wrong: A continuous trigger does keep the job perpetually active, but it burns compute through the many hours in which nothing has changed and it reacts to the passage of time rather than to a commit on the silver table.

Why C is wrong: The stated reason is wrong, since the table update trigger responds to data commits, and padding a cron time to cover the worst case is the guessing the team explicitly ruled out.

Why D is correct: The table update trigger watches one or more Unity Catalog tables for new commits and starts the job when an update lands, which couples the gold run to the silver load without a fixed clock time.

Free sampleWorking with Lakeflow Jobshard

An engineer converts an ingestion job from an hourly schedule to a file arrival trigger on its landing location. The task itself is a notebook whose read lists every object under the landing path and appends what it finds to a bronze table. After the change, the bronze table gains duplicate copies of files that earlier runs had already loaded. What does this reveal about the responsibilities of the trigger and the task?

  • AThe trigger moves the files it detected into a staging area for the run, so duplicates mean the notebook read the original landing path rather than the staging path the run was given.
  • BThe trigger decides when a run starts and does not tell the task which files caused it, so processing each file once remains the task's job and requires incremental ingestion such as Auto Loader or COPY INTO. Correct
  • CThe trigger fires once for each newly detected file, so the duplicates come from the several runs the burst produced, and lowering the number of concurrent runs the job allows removes them.
  • DThe trigger passes the list of newly arrived paths to the task automatically, so the duplicates mean the notebook ignored that supplied list and instead performed its own listing of the whole location.
A file arrival trigger only starts a run; exactly once file processing still depends on the incremental ingestion the task performs. Triggering and ingestion are separate concerns. The trigger observes the monitored location to decide that a run is worth starting, while the record of which files have already been consumed lives in the ingestion mechanism, such as an Auto Loader checkpoint or the load history COPY INTO maintains for its target table. A task that lists the whole path afresh has no such record and reloads everything.

Why A is wrong: A staging handover would be a tidy design and candidates who have built one by hand may expect it, which makes this plausible. It is wrong because the trigger does not relocate or copy any data, and the files stay where the producer wrote them.

Why B is correct: Correct. A file arrival trigger supplies a start signal derived from change in the monitored location, and it keeps no per file processing state on the task's behalf, so exactly once file handling still depends on the ingestion mechanism the task uses.

Why C is wrong: Concurrency is a real setting and a burst genuinely can produce more runs than expected, so the diagnosis feels concrete. It is wrong because repeated runs of a full directory listing duplicate history whatever their concurrency, so limiting parallelism does not address the cause.

Why D is wrong: It is the most seductive option, because the trigger clearly knows which files it saw and passing them on would be useful. It is wrong because no such list is handed to the task, so a notebook that reads the location itself is doing the only thing available to it.

Free sampleWorking with Lakeflow Jobshard

A Lakeflow Job has a task named extract that runs on serverless compute and records a watermark with the call below, and a dependent task named load that runs on a separate job cluster and reads it back. A colleague asks how the value crosses between two different compute resources and how long it lasts. Which statement is correct?

# in the extract task
dbutils.jobs.taskValues.set(key = "max_ts", value = watermark)
# in the load task
max_ts = dbutils.jobs.taskValues.get(taskKey = "extract", key = "max_ts")
  • AThe value lives in the driver memory of the compute that set it, so load can read it only when both tasks are configured to share one job cluster, and it is lost when that cluster terminates.
  • BThe jobs service holds the value for the duration of the run rather than in cluster memory, so load reads it on separate compute, and the value is scoped to that one run rather than carried into the next one. Correct
  • CThe value is persisted under the job identifier in the workspace file system, so a later run of the same job reads whatever the previous run left behind until a task overwrites it.
  • DThe value is visible to every task in the run irrespective of dependency, including tasks that started before extract, because the jobs service resolves the whole graph before any task begins.
Task values are stored by the jobs service, scoped to a single job run, and readable by downstream tasks even on different compute. A task value is recorded against the task run by the jobs service rather than kept in the memory of the compute that set it, so a downstream task on entirely separate compute can fetch it by task key and key name. Because the scope is the run, nothing set tonight is visible to tomorrow's run.

Why A is wrong: It is a reasonable guess for anything set through dbutils, but sharing compute is not a condition of reading a task value, and the value outlives the compute that produced it.

Why B is correct: Task values are recorded against the task run by the jobs service, which is why they survive the loss of the setting task's compute and why each run starts with its own set of values.

Why C is wrong: This treats task values as durable state and is the trap behind incremental jobs that silently reuse yesterday's watermark. Values do not carry across runs and a fresh run starts with none.

Why D is wrong: The graph is resolved up front, which makes this sound reasonable, but a value cannot be read before the task that sets it has produced it, so only downstream tasks can rely on it.

Other domains in this exam

See also the Data-Engineer-Associate cert hub, the study guide, and the cheat sheet.

Examworthy is not affiliated with or endorsed by Databricks. Original, blueprint-aligned practice material only.