A bronze Delta table in a Microsoft Fabric Lakehouse receives an hourly extract from a webhook source that occasionally redelivers the same event, so the same EventId can appear several times with identical column values. The team needs the silver table to keep exactly one row per EventId, and when an EventId recurs the row with the latest IngestedUtc should win. Using a Spark notebook, which approach best deduplicates the data?
# bronze rows: EventId, payload, IngestedUtc (duplicates possible)
# goal: one row per EventId, latest IngestedUtc wins- AApply dropDuplicates on the EventId column alone, which keeps one arbitrary row per EventId and discards the rest before writing to the silver table.
- BUse a window partitioned by EventId ordered by IngestedUtc descending, assign a row number, and keep only the rows where the row number equals one before writing. Correct
- CGroup by EventId and take the maximum IngestedUtc per key, then join that result back to the source on both EventId and IngestedUtc to select the winning row.
- DAppend every extract to the silver table and rely on Delta Lake to discard duplicate EventId values automatically because the table is stored in Delta format.
Why A is wrong: dropDuplicates on EventId removes duplicates but keeps an arbitrary surviving row, so it cannot guarantee the latest IngestedUtc wins; the tie-break the requirement states is left to chance, which is the trap here.
Why B is correct: Partitioning by EventId and ranking by IngestedUtc descending makes the latest row rank one for each key, and filtering to rank one keeps exactly one deterministic winning row per EventId in a single pass, which is what the requirement asks.
Why C is wrong: This returns the latest row per EventId, but if two redeliveries share the maximum IngestedUtc the join emits both, reintroducing duplicates; a window with a deterministic ordering is the cleaner single-pass answer.
Why D is wrong: Delta Lake does not enforce primary-key uniqueness, so appending simply stores every redelivered row; the format gives transactions and versioning, not automatic deduplication, so duplicates would accumulate.