NCA-ADS - Machine Learning With RAPIDS (16% of the exam) - Section 2.6
Interpret performance metrics.
Interpret classification and regression metrics including accuracy, precision, recall, F1, AUC-ROC, RMSE, and R-squared in the context of the problem. Choose the metric that best reflects business or scientific objectives, especially when class distribution is skewed.
Practice question for this objective
lock_openFree sampleMachine Learning With RAPIDSmedium
A fraud-detection model flags 90 fraudulent transactions out of 100 actual fraud cases, but also flags 810 legitimate transactions as fraud. What does this result tell you about precision and recall, and which metric is more appropriate to prioritise given that the cost of missing a fraud case is very high?
APrecision is 10% and recall is 90%. Because missing a fraud case is costly, recall should be prioritised.check_circle Correct
BPrecision is 90% and recall is 10%. Because missing a fraud case is costly, precision should be prioritised.
CPrecision is 10% and recall is 90%. Because so many legitimate transactions are incorrectly flagged, precision should be prioritised.
DPrecision is 90% and recall is 90%. Accuracy is the right summary metric because both classes perform equally well.
Compute and interpret precision and recall from a confusion matrix, and select the metric that aligns with a given business cost. Precision = TP / (TP + FP) captures how trustworthy a positive prediction is, while recall = TP / (TP + FN) captures how many real positives are found. These two metrics embody a trade-off: relaxing the decision threshold raises recall at the cost of precision. In high-cost false-negative contexts such as fraud detection, medical diagnosis, or safety systems, recall is the primary metric to optimise because missing a true case is more damaging than generating extra false alarms.
Why A is correct: Precision is TP / (TP + FP) = 90 / (90 + 810) = 10%, meaning nine in ten flagged items are false alarms. Recall is TP / (TP + FN) = 90 / 100 = 90%, meaning nearly all real fraud is caught. When false negatives carry severe cost, maximising recall is the correct business priority.
Why B is wrong: This inverts the computed values. Precision measures the fraction of flagged cases that are truly fraud, not the fraction of real frauds caught, so the 90-out-of-100 figure is recall, not precision.
Why C is wrong: The metric values are correct here, but the business logic is inverted. When missing a fraud case is very costly, recall matters more - the harm from false negatives outweighs the inconvenience of false positives in this scenario.
Why D is wrong: Neither value is correct. Accuracy conflates true positives and true negatives into a single number and is misleading when classes are imbalanced. The scenario shows very different error rates for the two classes, making precision and recall far more informative than accuracy.
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-ADS bank for this domain.
Model Y is strictly better because its RMSE is lower, and RMSE is always the more reliable regression metric regardless of the error distribution.
errorWhy it is wrong: Declaring one metric universally superior ignores the information the two metrics jointly convey. RMSE and MAE serve different purposes, and choosing between models requires understanding what type of error matters most in the business context.
0.68, because recall divides the 170 correctly predicted churners by the 250 transactions the model predicted as churn.
errorWhy it is wrong: This is tempting because 170 / (170 + 80) = 0.68 uses the right numerator, but that ratio is precision, not recall. Recall must divide by the actual positives (170 + 30), not by the total predicted positives.
Raise the threshold above 0.5, which tends to raise recall and lower precision for the positive class.
errorWhy it is wrong: Raising the threshold is tempting if one assumes a stricter cut-off catches more disease, but it labels fewer patients positive, so it lowers recall and tends to raise precision, the opposite of what missing cases requires.