An advertising platform processes a continuous stream of bid-request events from Pub/Sub at around 400,000 messages per second and must compute fraud features with millisecond-scale per-event latency. The team already operates a Dataproc cluster for nightly Spark SQL reports and is debating whether to extend that cluster to handle the real-time path using Spark Structured Streaming, or to build a separate streaming pipeline. Throughput, autoscaling responsiveness, and operational simplicity all matter. Which approach best fits the real-time path?
- AExtend the existing Dataproc cluster with a Spark Structured Streaming job that pulls from Pub/Sub using a community connector, sharing capacity with the nightly batch job.
- BCreate a separate Dataproc cluster running Spark Structured Streaming and rely on cluster autoscaling based on YARN pending memory to track demand.
- CRun a Cloud Data Fusion realtime pipeline that consumes from Pub/Sub and pushes transformations down to a Dataproc Spark Streaming runtime managed by Data Fusion.
- DBuild a Dataflow streaming pipeline using Apache Beam, consume from Pub/Sub with the native source, and rely on Streaming Engine and horizontal autoscaling to track the workload. Correct
Why A is wrong: Co-locating a sub-second streaming job with a heavy nightly batch on the same cluster causes resource contention, capacity sized for nightly peaks rather than streaming backlog, and noisy-neighbour failures, and Spark on Dataproc does not autoscale as responsively as Dataflow for streaming.
Why B is wrong: A dedicated Spark Streaming cluster removes the contention problem but still requires managing cluster shape, image, and node turnover, and Dataproc cluster autoscaling on YARN pending memory reacts on a minute scale rather than the sub-second responsiveness expected of a managed streaming runner.
Why C is wrong: Cloud Data Fusion targets visual ETL with modest throughput and low-code maintenance, and its realtime pipelines still execute on Dataproc under the hood, inheriting the same autoscaling and operational concerns without providing the latency profile a 400,000 messages per second fraud feature path needs.
Why D is correct: Dataflow with Streaming Engine is a managed runner purpose-built for high-throughput, low-latency event processing, autoscales workers based on backlog and CPU, integrates natively with Pub/Sub, and isolates the real-time path from the nightly batch cluster's operational risks.