A team builds a RAPIDS-based ETL and model-training pipeline. After six months, a re-run of the pipeline on the same raw data produces different model metrics. An audit finds that cuDF, cuML, and CUDA were all updated in the shared environment, and the Parquet input files were repartitioned. Which single pipeline reproducibility practice, if it had been followed from the start, would have been sufficient to catch this class of problem before the re-run?
- ASet a fixed random seed at the top of every pipeline script so that all stochastic operations produce the same draw sequence
- BEnsure that Dask workers process DataFrame partitions in deterministic order by sorting partition keys before any aggregation step
- CCapture the full pipeline configuration - including library versions, CUDA toolkit version, and a hash or snapshot of the versioned input data - in a config artefact stored alongside each pipeline run Correct
- DContainerise the pipeline with Docker so that the runtime environment is isolated from changes on the shared cluster
Why A is wrong: Fixing random seeds is necessary for stochastic reproducibility, but it does not detect or prevent changes caused by library version upgrades or repartitioned input data, which are the stated causes in this scenario.
Why B is wrong: Deterministic partition ordering prevents order-dependent aggregation variance and is a sound pipeline practice, but it cannot catch metric changes caused by library API changes or altered input files.
Why C is correct: Recording the complete configuration at run time creates a manifest that makes any deviation - version drift or data mutation - immediately visible when comparing a new run against the baseline, enabling diagnosis before results diverge silently.
Why D is wrong: Containerisation is a reproducible-environments strategy that would also have prevented the version drift, but the question asks about pipeline reproducibility practices, not environment isolation. A run-config artefact catches both version drift and data mutation regardless of whether a container is used.