A data team must bring records from a niche marketing SaaS application into Unity Catalog tables on a daily schedule. Lakeflow Connect offers no managed connector for that application, the vendor exposes only a paginated REST API, and the team has no capacity to build and maintain authentication, pagination and incremental change handling itself. The landed tables must still be governed and audited in Unity Catalog. Which approach fits the requirement?
- ARun an Auto Loader stream configured with a schema location that reads directly from the vendor REST API endpoint and appends the responses to a Unity Catalog managed table.
- BWrite a COPY INTO statement whose source location is the vendor REST API endpoint, and schedule it hourly as a Lakeflow Jobs task against a Unity Catalog managed table.
- CBuild a bespoke Python task in Lakeflow Jobs that calls the vendor API, handles paging and watermarks in its own state table, and writes the output with a Delta append.
- DUse Partner Connect to set up a validated ingestion partner tool that pulls the application through its own maintained connector and lands the results in Unity Catalog tables. Correct
Why A is wrong: Tempting because Auto Loader is the default incremental ingestion tool on Databricks, but it reads files from cloud object storage only, so it cannot call a REST API and has no way to page through one.
Why B is wrong: Tempting because COPY INTO is idempotent and simple to schedule, but its source must be a cloud object storage path or an external location, and it has no mechanism for calling a web service.
Why C is wrong: Tempting because it would technically work, but it puts the exact authentication, pagination and incremental logic the team said it cannot maintain back into code they own.
Why D is correct: Correct, because Partner Connect exists for exactly the sources Databricks does not ship a managed connector for, and the partner maintains the API handling while the landed tables remain governed by Unity Catalog.