A data engineer holds two DataFrames of the same width but with their columns declared in a different order: daily_north has columns order_id and region, while daily_south has columns region and order_id, both typed STRING. The engineer combines them with the call below and inspects the result. Which statement describes what the result contains?
combined = daily_north.union(daily_south)
combined.show(5)- AThe rows are aligned by column name, so every region value lands under region and every order_id value lands under order_id in the combined result.
- BThe call raises an analysis error, because the two DataFrames declare the same column names in a different order and Spark refuses to resolve the union.
- CThe rows are combined by column position, so the region values from daily_south land in the order_id column of the result and its order_id values land in region. Correct
- DThe rows are combined by column position and duplicate rows are then removed, so the result holds fewer rows than the two input DataFrames hold together.
Why A is wrong: This is tempting because SQL practitioners think of a union as a set operation over named columns, but the DataFrame union method resolves columns by position and ignores their names entirely.
Why B is wrong: An analysis error is a reasonable guess given how strict Spark is about schemas, but union checks only the number of columns and the compatibility of the types, which both hold here, so the query runs.
Why C is correct: Correct: DataFrame union is a positional operation, so the first column of the second DataFrame is stacked under the first column of the first, producing silently transposed values when the orders differ.
Why D is wrong: The positional half of this is right, but the DataFrame union method behaves like SQL UNION ALL and keeps duplicates; removing them requires an explicit call to distinct or dropDuplicates.