Data-Engineer-Associate - Data Transformation and Modeling (22% of the exam) - Section 3.7

Apply data quality checks and validation rules to ensure reliable Silver and Gold datasets.

Enforce data quality with pipeline expectations and table constraints, and choose the action on violation: keep and record the row, drop it, or fail the update. Recognise how quarantining bad records keeps Silver and Gold datasets reliable.

pipeline expectationsexpect or dropCHECK constraintsquarantine patterndata quality metrics

Practice question for this objective

Free sampleData Transformation and Modelingmedium

A gold dataset in a Lakeflow Spark Declarative Pipelines pipeline declares the expectation below. Tonight's update reads a batch in which one record carries a negative revenue value. What is the effect on the update and on the published table?

CONSTRAINT non_negative_revenue EXPECT (revenue >= 0) ON VIOLATION FAIL UPDATE

AThe update fails, so the table keeps the contents it held before the run, and a further update succeeds only once the source data or the expectation itself has been corrected. Correct
BThe violating record is discarded, the remaining records are published, and the pipeline is reported as degraded until an engineer acknowledges the warning in the interface.
CThe update runs to completion and the violating record is published with the rest of the batch, while the failure count is raised against the named expectation.
DThe violating record is held back in pipeline state and published automatically during a later update, once a corrected revenue value for that record arrives.

ON VIOLATION FAIL UPDATE aborts the whole update on any violating record, leaving the target table at its previous contents. The three actions available to an expectation differ in how far a violation propagates: warning publishes the record, dropping removes the record, and failing the update stops the run before anything is published. Because the update is atomic, the table is left at its last good state, which is what makes this action suitable for a rule that downstream consumers rely on absolutely.

Why A is correct: Correct. FAIL UPDATE treats a violation as fatal, so nothing from that update is published and the previous state of the table stands.

Why B is wrong: Discarding the record is the DROP ROW action, and there is no degraded state to acknowledge; FAIL UPDATE is deliberately stronger than either of those.

Why C is wrong: This is what an expectation with no action clause does; adding ON VIOLATION FAIL UPDATE is precisely the instruction not to publish data that breaks the rule.

Why D is wrong: No holding area for rejected records exists; a record kept for later inspection has to be routed to a quarantine dataset that the author declares.

See more Data-Engineer-Associate practice questions, answers explained.

Exam traps in Data Transformation and Modeling

Answers that look right on this material and are not. Each one is a distractor from a different question in the Data-Engineer-Associate bank for this domain.

It writes the 1,200 records to the target table and counts nothing against the expectation, because an expectation with no action clause is evaluated in development mode alone.
Why it is wrong: Tempting because expectations feel like a development aid, but the evaluation is part of every update in any mode, and the failing records are counted whichever mode the pipeline runs in.
A single expectation carrying ON VIOLATION FAIL UPDATE, so the run halts and an operator inspects the offending records in the source files.
Why it is wrong: Tempting because failing loudly does preserve the bad records, but halting every update on the first invalid row breaks the requirement that processing continues.
Declaring the expectation with ON VIOLATION FAIL UPDATE keeps the update running and diverts the failing rows into a side table for later review.
Why it is wrong: Tempting because it sounds like a safe middle ground, but the fail action stops the update immediately and no side table is produced by the expectation itself.

Examworthy is not affiliated with or endorsed by Databricks. Original, blueprint-aligned practice material only.