Data-Engineer-Associate - Data Transformation and Modeling (22% of the exam) - Section 3.5

Understand the basic tuning parameters (shuffle partitions, default parallelism, executor and driver memory, auto broadcast join threshold) and re-measure performance.

Explain what spark.sql.shuffle.partitions, spark.default.parallelism, executor and driver memory, and spark.sql.autoBroadcastJoinThreshold control, predict the effect of raising or lowering each, and re-measure after every change. Recognise where adaptive query execution already tunes these at runtime.

spark.sql.shuffle.partitionsspark.sql.autoBroadcastJoinThresholdexecutor memoryadaptive query executionre-measure after tuning

Practice question for this objective

Free sampleData Transformation and Modelinghard

A data engineer tunes a PySpark job that reads two Delta tables, joins them and aggregates the result, on a cluster with 64 cores in total. Wanting to control how many partitions the job's shuffles produce, the engineer finds both spark.sql.shuffle.partitions and spark.default.parallelism in the cluster's Spark configuration and asks which one governs this workload. Which two statements are correct for this job? (Select TWO.)

Aspark.default.parallelism takes precedence over spark.sql.shuffle.partitions whenever both are set, so the DataFrame join produces one partition per core on the cluster.
Bspark.sql.shuffle.partitions is fixed when the cluster starts, so changing the number of partitions for this job requires editing the cluster configuration and restarting the compute.
Cspark.sql.shuffle.partitions sets the number of partitions produced by shuffles in DataFrame and SQL operations, including this job's join and aggregation, and it defaults to 200. Correct
Dspark.default.parallelism sets the number of partitions produced by shuffles in DataFrame and SQL operations, so raising it is the supported way to split this job's join more finely.
Espark.default.parallelism governs the default partition count for RDD operations such as reduceByKey and parallelize, so it does not control the shuffles this DataFrame job performs. Correct

DataFrame and SQL shuffles read spark.sql.shuffle.partitions, while spark.default.parallelism applies to RDD operations. The two settings sit in different layers of the engine. The DataFrame and SQL planner decides post shuffle partitioning from spark.sql.shuffle.partitions, whose default is 200 regardless of cluster size, and that value is a session configuration changeable at runtime. spark.default.parallelism supplies a default partition count for RDD transformations and for parallelize, so it has no bearing on a job written entirely with DataFrames.

Why A is wrong: Tempting because the name reads like a global default, but there is no such precedence: the SQL and DataFrame planner reads its own shuffle partition setting and ignores this one.

Why B is wrong: Plausible because several cluster level settings do behave this way, but this is a session configuration that can be set at runtime from the notebook and takes effect on subsequent queries.

Why C is correct: This is precisely the setting the DataFrame and SQL engine consults when deciding how many post shuffle partitions a join or aggregation writes, and 200 is its documented default.

Why D is wrong: This mirrors the correct statement with the wrong setting named, which is exactly the confusion the two similar names create; the DataFrame planner does not read this value.

Why E is correct: The setting supplies the default partitioning for RDD level transformations and for parallelize, which leaves a DataFrame job like this one entirely unaffected by it.

See more Data-Engineer-Associate practice questions, answers explained.

Exam traps in Data Transformation and Modeling

Answers that look right on this material and are not. Each one is a distractor from a different question in the Data-Engineer-Associate bank for this domain.

A shuffle partition value above the number of cluster cores is discarded during planning, so the reading stage falls back to the total core count of the cluster for its task count.
Why it is wrong: This is tempting because task counts often settle near a multiple of the core count, but no rule discards a configured value for exceeding the core count, and a static plan would produce all 800 tasks regardless.
spark.default.parallelism sets the partition count produced by the join shuffle, while spark.sql.shuffle.partitions applies to lower level RDD operations such as reduceByKey.
Why it is wrong: This is tempting because the word parallelism sounds like the more general setting and therefore the more authoritative one, but the two roles are reversed: the SQL setting is the one that governs a DataFrame join shuffle.
Raising spark.executor.memory is the change that applies, because the collected rows are assembled and held in executor heap before a reference to them is handed to the driver process.
Why it is wrong: This is tempting because executor memory is the setting tuned most often, but the executors only produce and return their own partitions, and the failure reported by the driver would persist after enlarging them.

Examworthy is not affiliated with or endorsed by Databricks. Original, blueprint-aligned practice material only.