Data-Engineer-Associate - Working with Lakeflow Jobs (16% of the exam) - Section 4.2

Configure common tasks (notebook, SQL query, dashboard and pipeline tasks) and their dependencies using Lakeflow Jobs and its DAG-based task graph.

Build a multi-task job from notebook, SQL, dashboard and pipeline tasks, wire dependencies into a directed acyclic graph, and pass parameters and task values between tasks. Choose the right compute for each task.

multi-task jobstask dependenciespipeline taskjob parametersjob compute

Practice question for this objective

Free sampleWorking with Lakeflow Jobsmedium

A Lakeflow Job runs five notebook tasks that all need the same processing date for a given run. The team wants that value set in one place per run, without editing five task configurations whenever it changes, and wants each notebook to read it as an ordinary parameter. Which statement about job parameters is correct?

{{job.parameters.processing_date}}
  • AJob parameters are fixed at the moment the job definition is saved, so supplying a different processing date means editing the job definition itself rather than starting another run with a new value.
  • BJob parameters are scoped to the first task in the graph, so the remaining four tasks receive the processing date only if that task writes it out again for them to read.
  • CJob parameters apply only to pipeline tasks, so the five notebook tasks have to take the processing date from a widget default written into each notebook instead.
  • DJob parameters are defined once on the job, are available to every task in the run, and reach a task through a parameter value written with the {{job.parameters.processing_date}} reference syntax. Correct
Job parameters are declared once on the job, supplied per run, and referenced by any task using the job.parameters syntax. Job parameters are key and value pairs held on the job rather than on an individual task. A default value is stored with the job, a different value may be supplied when a run is triggered by hand or through the API, and every task in that run can reach the value by referencing it in its own parameter list. That is what removes the need to edit five task configurations, and it differs from task values, which are produced by a running task and read by a task placed downstream of it.

Why A is wrong: Tempting because the parameter keys are indeed part of the job definition, but the values carry defaults that can be replaced when a run is triggered, which is the whole point of parameterising a run.

Why B is wrong: This confuses job parameters with task values, which are written by one task and read by another; job parameters are attached to the run and need no relay task.

Why C is wrong: Plausible because notebooks do expose widgets, but job parameters are not restricted by task type, and a widget default hard codes the value in five places, which is the problem being solved.

Why D is correct: Correct: the parameter is declared at job level, a value may be supplied when the run starts, and each task picks it up by referencing it in its own parameters.

See more Data-Engineer-Associate practice questions, answers explained.

Exam traps in Working with Lakeflow Jobs

Answers that look right on this material and are not. Each one is a distractor from a different question in the Data-Engineer-Associate bank for this domain.

  • Merge the five notebooks into a single notebook task on one job cluster, so that the run creates compute once and the startup is paid a single time.

    Why it is wrong: It does cut startup to one occurrence, which is why it appeals, but collapsing the tasks discards the per task status and retry granularity the team asked to keep, and a late failure then reruns all five notebooks.

  • The value lives in the driver memory of the compute that set it, so load can read it only when both tasks are configured to share one job cluster, and it is lost when that cluster terminates.

    Why it is wrong: It is a reasonable guess for anything set through dbutils, but sharing compute is not a condition of reading a task value, and the value outlives the compute that produced it.

  • The condition task is evaluated on the job cluster shared by its neighbouring tasks, so moving the decision out of the notebook adds one more task competing for that compute on every run.

    Why it is wrong: It assumes every task in a job needs compute. The condition task is evaluated by the jobs service itself and does not occupy or start a cluster.

Examworthy is not affiliated with or endorsed by Databricks. Original, blueprint-aligned practice material only.