A team is building a continuously running pipeline that cleans, parses, and tokenises text records as they arrive from a streaming source, with the same transformation logic later needed for a large historical backfill. They want one programming model that runs both streaming and batch on a managed, autoscaling service. Which statement most accurately describes Dataflow's role here compared with Apache Spark?
- ADataflow can only process bounded batch data, so the streaming portion must be handled separately by Spark Structured Streaming before the backfill runs.
- BDataflow and Spark both require the engineer to write SQL queries only, so the tokenisation logic cannot be expressed as custom transform code in either system.
- CDataflow is an in-memory single-node library suited to small datasets, while Spark is the only one of the two that can distribute the parsing work across machines.
- DDataflow runs Apache Beam pipelines as a fully managed service with unified batch and streaming semantics and autoscaling, whereas a self-managed Spark cluster needs the team to provision and tune the workers themselves. Correct
Why A is wrong: It is tempting because batch backfills are mentioned, but it is wrong: Dataflow handles unbounded streaming and bounded batch with the same Beam pipeline, so there is no need to offload the streaming part to Spark.
Why B is wrong: This is tempting if you conflate these engines with a SQL warehouse, but it is wrong: both Beam pipelines on Dataflow and Spark jobs let you write custom code for transforms like tokenisation, not just SQL.
Why C is wrong: This is tempting because in-memory tools exist for small data, but it is wrong: Dataflow is a distributed managed service that scales across many workers, so it is not a single-node library and both engines distribute work.
Why D is correct: Dataflow executes Apache Beam, whose model treats batch and streaming as one API, and the service autoscales workers without manual cluster management; running Spark directly typically means the team sizes, scales, and tunes the cluster, which is the operational difference the scenario asks about.