A fraud classifier on a dataset with only 2 per cent positives reports a high ROC-AUC of 0.93, yet stakeholders complain that most transactions it flags turn out to be legitimate. The team wants an evaluation curve that reflects how the trade-off between catching fraud and raising false alarms behaves under this severe imbalance. Which curve is the most informative choice?
- AA precision-recall curve, because under heavy imbalance it directly exposes how precision falls as recall rises, a pattern that ROC-AUC can mask. Correct
- BA calibration curve plotting predicted probability against observed frequency to check whether the scores are well calibrated.
- CA learning curve plotting training and validation error against the number of training samples to diagnose under- or over-fitting.
- DA residual plot of prediction errors against fitted values to check for systematic bias across the prediction range.
Why A is correct: Correct. The precision-recall curve plots precision against recall across thresholds, so it directly reveals the high false-alarm rate at a given recall. Under severe imbalance ROC-AUC can stay high while precision is poor, which is exactly the stakeholders' complaint.
Why B is wrong: A calibration curve assesses probability reliability, not the precision-versus-recall trade-off the stakeholders care about. It is tempting because calibration is a valid diagnostic, but it does not show how flagging more fraud affects false alarms, so it is wrong.
Why C is wrong: A learning curve diagnoses bias and variance with respect to dataset size, not the threshold trade-off between precision and recall. It is tempting because it is a standard evaluation plot, but it does not address imbalance scoring, so it is wrong.
Why D is wrong: Residual plots are a regression diagnostic for continuous targets, not a classification imbalance tool. It is tempting because it sounds like an evaluation chart, but it does not apply to a fraud classifier's precision-recall behaviour, so it is wrong.