A gold Lakeflow Job aggregates a silver Unity Catalog table that three separate upstream jobs write to at different times of the morning, and their finishing times drift by up to ninety minutes. The gold job currently runs on a cron schedule chosen to sit after the slowest upstream job, so it usually reads stale data on fast mornings and occasionally reads a half loaded table on slow ones. The team cannot change the three upstream jobs. Which trigger should the gold job use?
- AA file arrival trigger on the cloud storage path that holds the silver table's underlying data files in the metastore.
- BA continuous trigger, so the gold aggregation is always rerunning and therefore always reflects the latest state of the silver table.
- CA scheduled trigger set to a much shorter interval, with a first task that exits early when the silver table shows no new version.
- DA table update trigger on the silver table, so the gold job starts a run when a new commit is made to that table. Correct
Why A is wrong: It sounds equivalent because a Delta write does add files, but the file arrival trigger is meant for a landing location of raw files and would fire on intermediate file activity rather than on a committed table version.
Why B is wrong: Continuous execution would keep the aggregate fresh, but it burns compute all day for a source updated three times a morning, and the team asked to react to upstream completion rather than to run without pause.
Why C is wrong: This is a workable but cause blind fix: it rebuilds trigger logic by hand inside the job and still starts many runs whose only purpose is to discover that nothing has changed.
Why D is correct: A table update trigger watches for commits to the named Unity Catalog table, which is the only signal the gold job can observe that reflects the upstream writes without touching the upstream jobs.