PDE domain - 25% of the exam

Ingesting and Processing Data

Ingesting and Processing Data is 25% of the PDE exam. These are the objectives it covers, each with practice questions, with every answer explained.

The domain in numbers

86
Practice questions
4
Objectives
58
Medium
28
Hard

Where people go wrong

  • Swapping event time for processing time

    The wrong answers move windowing onto processing time to sidestep late or skewed data. Processing-time windows place a reading in whichever window is current when it arrives, so per-window figures reflect Dataflow's delays and not the period the reading describes. The correct answers keep event-time windows and fix the cause: allowed lateness sized to the straggler delay, a trigger that fires again for late elements, or a timestamp taken from the event_time attribute.

    See it in a sample question
  • Reading Pub/Sub exactly-once as end-to-end deduplication

    The wrong answers treat Pub/Sub exactly-once delivery, or Beam exactly-once processing, as proof that no duplicate reaches BigQuery. Each guarantee covers only its own layer. Pub/Sub treats a producer that re-publishes the same match as a new message, and it does nothing about duplicate offsets from a Kafka replay. A deterministic per-event identifier, used as the BigQuery insertId or as the sink's deduplication key, closes that gap.

    See it in a sample question
  • Treating Cloud NAT or Private Google Access as a private path everywhere

    The wrong answers pick Cloud NAT or Private Google Access whenever workers have no external IP. Cloud NAT sends traffic out through public IPs, so it cannot keep API calls off the internet or inside the perimeter. Private Google Access lets a subnet reach Google APIs, but it gives no route to a Cloud SQL private IP, which needs a private services access peering, and it does not cover on-premises hosts, which need Private Service Connect for Google APIs over the Interconnect.

    See it in a sample question
  • Offering Google-managed or customer-supplied keys where CMEK is required

    The wrong answers offer Google-managed keys or customer-supplied keys when the team must rotate, disable or revoke its own key. Google-managed keys cannot be revoked by the team on demand. Customer-supplied keys are accepted only by specific services such as Cloud Storage and Compute Engine, not by Pub/Sub, Dataflow state or BigQuery, and they make the team handle raw key material. Customer-managed keys in Cloud KMS cover all three, and Cloud External Key Manager applies when key material must stay in an external HSM.

    See it in a sample question
  • Rewriting a Spark workload in Beam

    The wrong answers move a Spark job to Dataflow to gain managed operation or service-side shuffle. Dataflow runs Beam, not Spark, so this means rewriting the job, which is heavy work for a team with no Java engineers and PySpark code that pins pandas or NumPy versions. The correct answers stay on Spark: Dataproc Serverless with a custom container, or a Dataproc cluster with standard primary workers, a mixed secondary pool and enhanced flexibility mode.

    See it in a sample question

Objectives in this domain

What to study

The official documents this domain's practice questions cite most.

Sample questions from this domain

Free sampleIngesting and Processing Datahard

A logistics team runs a Dataflow streaming job that computes per-minute delivery counts in tumbling windows. Roughly 2 percent of GPS events arrive between 30 seconds and 8 minutes after their event time because of intermittent driver connectivity, and downstream dashboards must reflect corrected counts when late data lands. The job currently uses the default trigger with no allowed lateness, and late events are being dropped. Which configuration best preserves accuracy while keeping per-window state bounded?

  • ASwitch to processing-time windows of one minute so events are bucketed on arrival and lateness becomes irrelevant.
  • BKeep the default trigger and set allowed lateness to 10 minutes, accepting that late panes will overwrite earlier results in the sink.
  • CConfigure an event-time trigger at the end of the window plus a late-firing trigger after each late element, with allowed lateness of 10 minutes and accumulating panes. Correct
  • DDisable the watermark by setting allowed lateness to an unlimited duration and rely on a global trigger to fire once at job drain.
Combine event-time triggers, late-firing triggers, and accumulating panes to handle bounded late data without unbounded state. The watermark estimates the progress of event time and gates the on-time pane for a window. Configuring a late-firing trigger together with bounded allowed lateness keeps per-window state alive only as long as late data may reasonably arrive, while accumulating panes mean each emission represents the full corrected count for the window. This is the textbook pattern for dashboards that must converge to an accurate event-time result.

Why A is wrong: Processing-time windows mis-attribute late events to the window in which they happen to arrive, which corrupts per-minute delivery counts and is the opposite of what the dashboard requires for event-time correctness.

Why B is wrong: The default trigger fires once at the end of the window and again per late element, but without an explicit late trigger the late-data semantics are implementation-defined and discarding state behaviour cannot be coordinated with accumulation mode for the dashboard.

Why C is correct: An event-time trigger emits an on-time result when the watermark passes the window end, while a late trigger emits an updated pane for each late element within the 10-minute allowance, and accumulating panes ensure each emission represents the corrected cumulative count for the window.

Why D is wrong: Unlimited allowed lateness causes per-window state to grow without bound, and waiting for drain defeats the purpose of streaming dashboards by delaying every result until the job stops.

Free sampleIngesting and Processing Datahard

A news media team consumes article view events from a self-managed Kafka cluster into a Dataflow streaming pipeline that computes top stories per region in five-minute sliding windows. Replays for backfill occasionally cause the same offsets to be delivered twice, and the downstream BigQuery sink must not double-count views. The team is comparing Pub/Sub exactly-once delivery, Beam's exactly-once processing semantics, and idempotent writes keyed on a per-event identifier. Which approach most accurately reflects how Beam achieves the team's end-to-end guarantee from Kafka into BigQuery?

  • ACombine the KafkaIO source's exactly-once read configuration with a deterministic per-event identifier carried through the pipeline and use BigQuery streaming inserts with that identifier as the insertId to deduplicate at the sink. Correct
  • BBeam's exactly-once processing semantics extend transparently to the Kafka source, so simply enabling the Dataflow runner is sufficient to guarantee exactly-once view counts in BigQuery without further configuration.
  • CSwitching the source from Kafka to Pub/Sub with exactly-once delivery enabled removes the duplicate problem entirely, so the BigQuery sink can use streaming inserts without any deduplication key.
  • DUse the at-least-once Kafka source and rely on BigQuery's MERGE statement executed by a Dataflow side input every minute to remove duplicates after they have been written to the table.
Achieve end-to-end exactly-once from Kafka into BigQuery by combining source-level exactly-once reads with deterministic identifiers used as the BigQuery streaming insertId. End-to-end exactly-once in Beam is the product of three guarantees: a source that does not emit duplicates on retry, a runner with exactly-once processing for internal state, and a sink that can deduplicate on a stable key. KafkaIO supports exactly-once reads via offset tracking, and BigQuery streaming inserts deduplicate on insertId within a short window, so carrying a deterministic event identifier from source to sink closes the loop without after-the-fact reconciliation.

Why A is correct: Exactly-once read configuration prevents the source from emitting duplicates during runner-driven retries, while propagating a deterministic event identifier as the BigQuery insertId allows the sink to deduplicate replays that originate from outside the pipeline within BigQuery's best-effort window.

Why B is wrong: This conflates internal processing semantics with end-to-end delivery; Beam's exactly-once applies to the pipeline's own state, but a Kafka source replay introduces duplicates at the source boundary that the runner cannot deduplicate on its own.

Why C is wrong: Pub/Sub exactly-once removes a class of duplicates introduced by the messaging layer, but it does not address duplicate offsets created by Kafka replays, and the question is about the existing Kafka source rather than a wholesale migration.

Why D is wrong: A periodic MERGE looks pragmatic, but it processes duplicates after they are already visible to consumers, breaks the streaming freshness contract, and depends on side-input timing rather than a delivery-level guarantee at the sink.

Free sampleIngesting and Processing Datamedium

A Dataflow streaming pipeline running in a region must read from a Pub/Sub subscription and write to BigQuery without sending traffic over the public internet, in line with the organisation's networking policy. The pipeline runs in a Shared VPC subnet, and the security team has disabled external IP addresses on all Dataflow workers. Which combination of networking choices enables the workers to reach Pub/Sub and BigQuery while honouring this policy?

  • AAttach Cloud NAT to the worker subnet so the workers can reach Pub/Sub and BigQuery through a managed network address translation gateway.
  • BEnable Private Google Access on the subnet hosting the Dataflow workers so that requests to Pub/Sub and BigQuery resolve to private endpoints over Google's network. Correct
  • CPeer the Shared VPC to a Pub/Sub-managed VPC and to a BigQuery-managed VPC using VPC Network Peering so the worker subnet routes traffic directly to those services.
  • DUse Cloud VPN to tunnel traffic from the worker subnet to a Pub/Sub and BigQuery on-premises gateway that proxies API calls into Google's network.
Use Private Google Access on the Dataflow worker subnet so workers without external IPs can reach Pub/Sub and BigQuery APIs over Google's private network. Dataflow workers reach Pub/Sub and BigQuery as Google APIs. With external IPs disabled, the subnet feature Private Google Access lets the workers resolve those APIs to private addresses and reach them over Google's internal network, which satisfies a no-public-internet networking policy without introducing translation gateways or hybrid tunnels.

Why A is wrong: Tempting because Cloud NAT is the standard pattern for outbound internet from instances without external IPs, but traffic still leaves the Google network as public internet traffic to API endpoints, which violates the no-public-internet policy.

Why B is correct: Correct. Private Google Access on the worker subnet allows instances without external IPs to reach Google APIs, including Pub/Sub and BigQuery, over private addressing without traversing the public internet.

Why C is wrong: Plausible because peering is real, but Pub/Sub and BigQuery are not exposed as peer VPCs that customers can connect to; they are reached as Google APIs, so peering does not apply here.

Why D is wrong: Confuses hybrid connectivity with API access. Cloud VPN connects on-premises networks to a VPC; it does not change how Dataflow workers reach managed Google services and does not provide a private path to Pub/Sub or BigQuery.

Free sampleIngesting and Processing Datamedium

A regulated lender is planning a Dataflow streaming pipeline that reads from a Pub/Sub subscription and writes to BigQuery. The security team has disabled external IP addresses on all Dataflow workers and requires that worker traffic to Pub/Sub and BigQuery never traverse the public internet. Compliance also requires that data at rest is encrypted with keys the team can rotate and disable on demand, without the team handling raw key material on the workers. Which TWO planning decisions, taken together, satisfy both the networking and the encryption requirements? (Select TWO.)

  • AGive each Dataflow worker an external IP address but restrict egress with a firewall rule that allows only the published Google API address ranges, so workers reach Pub/Sub and BigQuery directly over a controlled path.
  • BEncrypt the BigQuery datasets with customer-supplied encryption keys passed on each API call, so the team holds the raw key bytes and supplies them to BigQuery whenever data is written or read.
  • CEnable Private Google Access on the workers' subnet and run the job with no public IP addresses, so workers reach the Pub/Sub and BigQuery APIs over internal addresses that route to Google APIs without internet egress. Correct
  • DConfigure customer-managed encryption keys in Cloud KMS for Pub/Sub, Dataflow, and BigQuery, granting the relevant service agents permission to encrypt and decrypt so the team can rotate or disable the keys on demand. Correct
  • ERely on Google-managed default encryption for Pub/Sub, Dataflow, and BigQuery, since data at rest is always encrypted, and document the default keys in the compliance evidence to show the data is protected.
Combine Private Google Access for no-external-IP workers with customer-managed encryption keys to satisfy private connectivity and controllable at-rest encryption together. Private Google Access is the mechanism that lets workers without external IPs reach Google API endpoints over an internal path, removing public internet egress, while customer-managed encryption keys in Cloud KMS give the team rotation and disable control over at-rest encryption across Pub/Sub, Dataflow, and BigQuery without ever placing raw key bytes on the workers. The two decisions address the networking and encryption constraints respectively.

Why A is wrong: This is tempting because allowlisting Google API ranges feels like a tight egress control, but assigning external IPs directly violates the stated no-external-IP policy and still sends traffic out through the internet edge rather than over a private path.

Why B is wrong: Customer-supplied encryption keys do let the team control the key, but they require the team to handle and pass raw key material on every operation, which contradicts the requirement to avoid managing raw keys on the workers, and BigQuery managed storage uses Cloud KMS keys rather than per-call supplied keys.

Why C is correct: Private Google Access lets instances with no external IP reach Google API endpoints over the internal network path, which is exactly what keeps the no-external-IP Dataflow workers able to call Pub/Sub and BigQuery without traversing the public internet.

Why D is correct: Customer-managed encryption keys in Cloud KMS let the team rotate and disable keys on demand while the services perform encryption and decryption through their service agents, so no raw key material is handled on the workers, which meets the encryption requirement across all three stages.

Why E is wrong: Default encryption does protect data at rest, but the keys are managed entirely by Google and cannot be rotated or disabled by the team on demand, so it fails the explicit requirement for keys the team controls.

Free sampleIngesting and Processing Datahard

A genomics research group runs a quarterly Spark batch job that scans 18 TB of compressed Parquet stored in Cloud Storage, performs heavy joins and aggregations, and writes results back to Cloud Storage. The job runs for around six hours once every three months and is otherwise idle. The team wants the lowest steady-state cost, no cluster management between runs, and the freedom to choose specific Spark and Python versions to match the libraries packaged with the analysis code. Which managed service approach best fits these constraints?

  • AProvision a long-running Dataproc standard cluster sized for peak load and rely on graceful decommissioning between quarterly runs to control cost.
  • BRewrite the job as an Apache Beam pipeline and submit it to Dataflow with the runner v2 in batch mode, using a custom container that pins the required Spark and Python versions.
  • CSubmit the workload as a Dataproc Serverless for Spark batch job, supplying a custom container image that pins the required Spark and Python versions and reading and writing directly from Cloud Storage. Correct
  • DCreate a Dataproc on Compute Engine ephemeral cluster per run with a workflow template, sized using only preemptible secondary workers, and delete the cluster on completion.
Choose Dataproc Serverless for Spark for infrequent, version-pinned PySpark batch workloads that read and write Cloud Storage. Dataproc Serverless for Spark is designed for fully managed, ephemeral PySpark and Spark SQL batches that bill only for the duration of execution, with no cluster to size or shut down. Custom container images let the team pin Spark and Python versions to match library dependencies, and reading directly from Cloud Storage avoids the need for persistent HDFS. For a job that runs six hours once per quarter and is otherwise idle, this gives the lowest steady-state cost and the least operational overhead while preserving the existing Spark code base.

Why A is wrong: Keeping a sized standard cluster running continuously is tempting because it removes start-up latency, but charging for compute that sits idle for almost three months between runs is the opposite of the lowest steady-state cost the team asked for.

Why B is wrong: Dataflow runs Beam, not Spark, so a custom container cannot make it execute pinned Spark code. Rewriting an 18 TB Spark job in Beam is a major engineering effort that the question does not justify and abandons the existing PySpark logic.

Why C is correct: Dataproc Serverless for Spark provisions ephemeral capacity per batch, charges only while the job runs, removes cluster ops between quarterly runs, and accepts a custom container image to pin Spark and Python versions, which matches every stated constraint.

Why D is wrong: An ephemeral Dataproc cluster does scale to zero between runs, but the team still owns cluster shape, image, and lifecycle tuning, and an all-preemptible secondary pool risks repeated task loss on a six-hour heavy-join job, making it less suitable than a serverless submission.

Other domains in this exam

See also the PDE cert hub, the study guide, and the cheat sheet.

Examworthy is not affiliated with or endorsed by Google Cloud. Original, blueprint-aligned practice material only.