NCA-ADS - Data Science Pipelines and Workflow Automation (13% of the exam) - Section 3.1
Design end-to-end data science pipelines.
Design end-to-end data science pipelines that sequence ingestion, preprocessing, training, evaluation, and deployment into a coherent workflow. Recognise where GPU acceleration can be inserted to reduce the total wall-clock time of the pipeline.
Practice question for this objective
lock_openFree sampleData Science Pipelines and Workflow Automationmedium
A team designs a five-stage batch pipeline: raw data ingestion, cleaning, feature engineering, model training, and evaluation. They want each stage to be independently testable and reusable across multiple projects. Which structural principle best supports that goal?
AImplement each stage as a self-contained, independently runnable unit with defined inputs and outputs, and compose stages through a common pipeline interfacecheck_circle Correct
BMerge cleaning and feature engineering into a single monolithic function to reduce the number of data handoffs between stages
CStore all intermediate outputs in memory across stages rather than persisting them, so the pipeline runs faster without disk overhead
DHard-code the data paths and model hyperparameters inside each stage function to ensure the pipeline produces reproducible results on every run
Identify the design principle that makes pipeline stages independently testable and reusable across end-to-end data science workflows. Modular pipeline design requires that each stage - ingestion, cleaning, feature engineering, training, evaluation - operates as a self-contained unit with clearly defined inputs and outputs. When stages expose a consistent interface and carry no hidden side effects or hard-coded dependencies, they can be tested individually with controlled synthetic data and composed freely in different pipeline configurations. This is the basis for maintainable, production-ready ML pipelines and aligns with the end-to-end pipeline design objective in the NCA-ADS blueprint.
Why A is correct: Modular stages with well-defined interfaces can be unit-tested in isolation using synthetic inputs, and the same stage can be dropped into different pipeline compositions without modification. This is the foundational design principle behind reusable, maintainable data science pipelines.
Why B is wrong: Consolidating stages into a monolithic function couples distinct concerns, making each part harder to test in isolation and impossible to reuse independently in other pipelines that may need only cleaning or only feature engineering.
Why C is wrong: In-memory chaining without stage boundaries prevents independent testing of any single stage because the full pipeline must always run from the start, and it makes individual stages impossible to reuse without re-executing earlier steps.
Why D is wrong: Hard-coding paths and hyperparameters within stage functions makes each stage tightly coupled to a specific environment and configuration, which prevents reuse across projects and makes testing in alternative environments impractical.
Exam traps in Data Science Pipelines and Workflow Automation
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-ADS bank for this domain.
Append each run's output to the existing result table so that no previously written data is ever lost between successive runs of the pipeline.
errorWhy it is wrong: Appending feels safe because it never deletes prior work, but on a re-run it duplicates rows, so two executions on the same input no longer produce the same downstream state, which is exactly the failure the requirement forbids.
Fit the imputer on the full dataset before splitting, then fit the scaler on the training split only, and store both as separate objects
errorWhy it is wrong: Fitting the imputer on the full dataset exposes validation and test values to the imputer during training, which constitutes target leakage through the preprocessing step and invalidates held-out evaluation.
Document the imputation and scaling steps so the serving team can re-implement the same logic and fit fresh parameters on the production data at inference time.
errorWhy it is wrong: Documentation aids understanding, but re-fitting parameters on production data produces different statistics from training, which is precisely the skew the design is meant to avoid.