A data engineer ingests JSON files that land continuously in a single cloud storage directory. That directory already holds several million historical files, and the ingestion latency has grown steadily as the file count has risen, with most of each run spent before any data is read. The engineer must keep ingestion incremental and reduce the discovery cost as the directory keeps growing. Which approach addresses the cause?
- AConfigure Auto Loader in directory listing mode so that each micro-batch enumerates the input directory and picks up files added since the previous run.
- BSchedule a COPY INTO statement every ten minutes so that the target table is refreshed from the same directory without a streaming query holding compute.
- CConfigure Auto Loader in file notification mode so that arriving files are discovered from cloud storage notification events instead of by enumerating the directory. Correct
- DRead the directory with spark.read.json on a scheduled Lakeflow Job and overwrite the target table so that the most recent files are represented in it.
Why A is wrong: Directory listing mode is a valid Auto Loader file discovery mode and is the default, which makes it tempting, but it is the mode already causing the problem because its cost scales with the number of files in the directory.
Why B is wrong: COPY INTO is a genuine incremental ingestion command and skips files it has already loaded, but it still discovers candidate files by listing the source directory, so the listing cost that dominates the run is unchanged.
Why C is correct: File notification mode subscribes to storage events for the input path, so discovery cost depends on how many files arrive rather than on how many files the directory already contains.
Why D is wrong: This runs on the current orchestration product and looks simple, but a full read and overwrite reprocesses every historical file on each run, which abandons incremental loading and makes the listing problem worse.