A team designs a five-stage batch pipeline: raw data ingestion, cleaning, feature engineering, model training, and evaluation. They want each stage to be independently testable and reusable across multiple projects. Which structural principle best supports that goal?
- AImplement each stage as a self-contained, independently runnable unit with defined inputs and outputs, and compose stages through a common pipeline interface Correct
- BMerge cleaning and feature engineering into a single monolithic function to reduce the number of data handoffs between stages
- CStore all intermediate outputs in memory across stages rather than persisting them, so the pipeline runs faster without disk overhead
- DHard-code the data paths and model hyperparameters inside each stage function to ensure the pipeline produces reproducible results on every run
Why A is correct: Modular stages with well-defined interfaces can be unit-tested in isolation using synthetic inputs, and the same stage can be dropped into different pipeline compositions without modification. This is the foundational design principle behind reusable, maintainable data science pipelines.
Why B is wrong: Consolidating stages into a monolithic function couples distinct concerns, making each part harder to test in isolation and impossible to reuse independently in other pipelines that may need only cleaning or only feature engineering.
Why C is wrong: In-memory chaining without stage boundaries prevents independent testing of any single stage because the full pipeline must always run from the start, and it makes individual stages impossible to reuse without re-executing earlier steps.
Why D is wrong: Hard-coding paths and hyperparameters within stage functions makes each stage tightly coupled to a specific environment and configuration, which prevents reuse across projects and makes testing in alternative environments impractical.