A data team must run a recurring job that reads several large catalogued data sets from Amazon S3, joins and cleanses them with Apache Spark, and writes the partitioned Parquet output back to S3 for analytics. The team wants a serverless service that scales Spark capacity automatically per run and integrates with the AWS Glue Data Catalog, so they neither size nor patch a cluster between runs. Which service best fits this transformation workload?
- AProvision a long-running Amazon EMR cluster with Apache Spark, schedule the join and cleansing steps on it, and keep the cluster available between runs so the recurring transformation always has capacity.
- BUse Amazon Athena CREATE TABLE AS SELECT statements to read the catalogued data, apply the joins and cleansing in SQL, and write partitioned Parquet results back into the analytics S3 location.
- CUse an AWS Glue Spark ETL job that reads the catalogued sources, performs the join and cleansing transformations, and writes partitioned Parquet to S3, with Glue provisioning the Spark capacity per run. Correct
- DBuild an AWS Lambda function that loads each S3 data set into memory, runs the join and cleansing in code on every schedule, and writes the partitioned Parquet output back to the analytics bucket.
Why A is wrong: EMR with Spark can perform the transformation, but a long-running cluster means the team sizes, patches, and pays for idle capacity between runs, which is the operational overhead they want to avoid.
Why B is wrong: Athena CTAS can transform and write Parquet, but it is a SQL query engine without the Spark programming model the team specifies and is less suited to complex multi-source ETL pipelines.
Why C is correct: Glue runs serverless Spark ETL jobs that read from and write to the Glue Data Catalog and S3, scaling worker capacity per run, so the team performs the join and cleansing without sizing or patching any cluster.
Why D is wrong: Lambda has tight memory and runtime limits and no native Spark engine, so joining several large data sets in a single function is impractical and would not scale to the volumes described.