A data engineer calls a paginated vendor REST API from a notebook and must land each page of raw JSON in cloud storage before any parsing, so that a later task can reprocess the untouched payloads. The landing area has to be governed and audited by Unity Catalog alongside the team's tables, and it must remain readable by a Lakeflow Job that runs the next morning on different compute. Where should the notebook write the raw files?
- ATo a path inside a Unity Catalog volume, for example /Volumes/main/raw/api_landing, so the files are governed by the metastore. Correct
- BTo the driver node's local file system under /tmp, and then register that directory as an external location on the same path.
- CTo the DBFS root of the workspace, which every cluster in that workspace can reach without any further configuration.
- DTo the cluster log delivery destination configured on the job compute, which keeps its contents after the compute terminates.
Why A is correct: A volume is the Unity Catalog object designed to hold non-tabular files, so the payloads inherit metastore permissions and audit logging and any later compute can reach the same path by name.
Why B is wrong: Writing with ordinary Python file calls does land bytes in /tmp, which makes this look workable, but that storage belongs to the driver instance and disappears when the compute terminates, and an external location must point at cloud object storage.
Why C is wrong: The DBFS root is reachable and needs no setup, which is why it is tempting, but it sits outside Unity Catalog, so the files carry no metastore permissions and no governed audit record.
Why D is wrong: Log delivery does persist beyond the life of the compute, so the durability requirement appears satisfied, but it is a diagnostics destination for driver and executor logs rather than a governed data landing area.