A Data Factory pipeline in Microsoft Fabric must copy daily CSV files from a third-party Amazon S3 bucket into a Lakehouse, then run a Spark notebook that cleans the data only after every file for the day has landed. Which orchestration keeps the steps ordered with the least custom code?
- ASchedule the Copy activity and the Spark notebook as two independent pipelines triggered at fixed times spaced far enough apart to avoid overlap.
- BHave the Spark notebook poll the Lakehouse folder in a loop and begin cleaning once it detects that the expected file count has arrived.
- CConfigure a Data Activator alert on the Lakehouse table that starts the Spark notebook whenever new rows are detected after the copy.
- DPlace the Copy activity and the notebook activity in one pipeline and connect them with an on-success dependency so the notebook runs after the copy. Correct
Why A is wrong: Two time-spaced schedules seem simple, but they only guess at completion timing and break whenever the copy runs long, so ordering is not actually guaranteed.
Why B is wrong: Polling from inside the notebook would work, but it pushes orchestration into custom code instead of using the pipeline dependency the platform already provides.
Why C is wrong: Data Activator reacts to data conditions, but wiring it to trigger the notebook adds an event layer that is heavier than a direct in-pipeline dependency.
Why D is correct: Chaining the notebook to the Copy activity with an on-success dependency enforces ordering natively in the pipeline without any extra code or external coordination.