Data Science Pipelines and Workflow Automation

A pipeline was written with the input bucket path, output path, model hyperparameters, and the GPU device count hard-coded as literals inside the stage functions. The team now needs to run the identical pipeline against a small sample in a development environment and against the full dataset in production, and they want a clean way to promote the same code between the two without editing source each time. Which design change most directly supports running one codebase across both environments?

• ADuplicate the pipeline into a development copy and a production copy, each carrying its own hard-coded literals, and keep the two in sync by hand when logic changes.
• BAdd conditional branches throughout the code that check the machine hostname and choose the development or production literals based on which host the pipeline detects it is running on.
• CExternalise the environment-specific values into configuration supplied at run time, so the same code reads its paths, hyperparameters, and device count from a config file or environment variables per environment.check_circle Correct
• DMove the hard-coded literals to the top of each stage function as module-level constants so they are at least grouped together and easier to locate before each run.

Why A is wrong: Two copies let each environment keep its own values, which appears to solve the immediate need, but it splits one pipeline into two divergent codebases that drift apart and must be reconciled manually on every change.

Why B is wrong: Hostname branching keeps everything in one file and switches values automatically, which seems convenient, but it bakes both environments' settings into the source, scatters fragile conditionals, and breaks the moment a new environment or host appears.

Why C is correct: Lifting the paths, hyperparameters, and device count out of the source into run-time configuration means a single codebase behaves differently per environment purely through the config it is given, which is exactly what lets the same code be promoted unchanged from development to production.

Why D is wrong: Grouping the literals as constants improves readability, which is a minor gain, but the values still live in source and must be edited and redeployed to switch environments, so it does not enable running one codebase across both.

Before trusting a quarterly re-run of a RAPIDS pipeline, a team wants an automatic check that the exact input file consumed is byte-for-byte identical to the file used in the baseline run, so that any silent change to the data is detected before metrics are compared. Which approach provides this guarantee most directly?

• ARecord a cryptographic content hash of the input file at run time and assert it equals the baseline hash before the pipeline proceeds.check_circle Correct
• BCompare the file's last-modified timestamp against the timestamp recorded for the baseline run and proceed only if they match.
• CCompare the file size in bytes against the size recorded for the baseline run and treat a match as confirmation the data is unchanged.
• DCompare the row count of the loaded DataFrame against the baseline row count and continue only when the counts agree.

Why A is correct: A content hash is derived from the file's bytes, so any change of any size produces a different digest; asserting the new hash equals the stored baseline proves the input is byte-for-byte identical.

Why B is wrong: Timestamps are easy to read, but they can be reset by copies or restores while the bytes differ, and can stay unchanged when content is edited in place, so they do not reliably prove the data is identical.

Why C is wrong: Equal file sizes are necessary but not sufficient, because two different files can share the same size, so a size match does not prove the contents are identical.

Why D is wrong: A row-count match catches added or removed rows, but it misses in-place edits to values within existing rows, so it cannot guarantee the data is byte-for-byte the same.

A team trains a cuML logistic regression on a dataset with 800 features and 3,000 rows. Training accuracy is very high but validation accuracy is much lower, and they suspect the model is fitting noise across the many correlated features. They want a single change that both shrinks coefficients and drives some of them to exactly zero, performing feature selection as part of fitting. Which approach satisfies both requirements?

• AApply L2 (ridge) regularisation, which adds a squared-magnitude penalty that shrinks all coefficients smoothly towards zero.
• BRemove the regularisation penalty entirely and instead increase the number of training iterations so the optimiser converges more fully.
• CIncrease the inverse-regularisation strength so the penalty becomes very weak, letting the model use every feature freely.
• DApply L1 (lasso) regularisation, whose absolute-value penalty shrinks coefficients and forces some of them to exactly zero, removing those features from the model.check_circle Correct

Why A is wrong: L2 regularisation does shrink coefficients and reduce variance, which makes it tempting, but its squared penalty pushes coefficients close to zero without ever setting them exactly to zero, so it does not perform the feature selection the team wants.

Why B is wrong: More iterations help the optimiser converge but do nothing to control variance, and with no penalty the model will fit the noise even more closely, widening the train-validation gap rather than performing any feature selection.

Why C is wrong: Weakening the penalty is the wrong direction, since a near-zero penalty lets coefficients grow to fit noise across all 800 features, which deepens the overfitting and selects nothing, even though loosening a constraint can sound like it improves fit.

Why D is correct: The L1 penalty's absolute-value geometry produces sparse solutions, so it both shrinks coefficients and sets a subset to exactly zero, achieving regularisation and embedded feature selection in one fit, which is precisely what this high-dimensional case needs.

A data scientist is laying out an end-to-end pipeline with five stages: raw ingestion, cleaning, feature engineering, model training, and a final report that compares the new model against the current production model. Cleaning depends on ingestion, feature engineering depends on cleaning, training depends on features, and the report depends on both training and the existing production metrics. The team wants the orchestrator to run each stage only after every stage it depends on has finished successfully, while still running independent stages concurrently where possible. Which way of expressing the pipeline most directly supports this execution behaviour?

• AA fixed list of stages that the orchestrator executes strictly top to bottom, so each stage always waits for the one written immediately above it regardless of what it actually depends on.
• BA single function that calls each stage in turn and relies on Python raising an exception to halt the remaining calls whenever an earlier stage fails partway through.
• CA directed acyclic graph in which each stage is a node and each dependency is an edge, so the orchestrator can schedule a stage once all of its upstream nodes succeed and run independent branches in parallel.check_circle Correct
• DA timed schedule that starts each stage at a fixed offset chosen to be long enough that the stages it depends on will normally have finished by then.

Why A is wrong: A linear ordered list does guarantee predecessors finish first, which looks sufficient, but it cannot express that two independent stages may run concurrently and it forces an order even where no real dependency exists, so it does not match the stated requirement.

Why B is wrong: Sequential function calls with exception propagation will stop on failure, which seems to enforce ordering, but it hard-codes one serial path, exposes no dependency structure to the orchestrator, and cannot run independent stages in parallel.

Why C is correct: Modelling stages as nodes and dependencies as edges in a directed acyclic graph lets the orchestrator derive the correct execution order from the edges, gate each stage on its upstream successes, and run branches with no dependency between them concurrently, which is exactly the required behaviour.

Why D is wrong: Staggered fixed-time starts can usually let predecessors finish first, which is tempting for simplicity, but timing is not a real dependency, so a slow upstream stage can cause a downstream stage to start on incomplete or missing data.