NCA-ADS - Descriptive Analysis and Visualization (13% of the exam) - Section 4.3
Apply hypothesis testing.
Apply hypothesis tests such as t-tests and chi-squared tests to determine whether observed differences between groups are statistically significant. Interpret p-values and confidence intervals correctly, and recognise the distinction between statistical significance and practical importance.
Practice question for this objective
lock_openFree sampleDescriptive Analysis and Visualizationmedium
A data scientist obtains a statistically significant result with a p-value of 0.001 when comparing mean session durations between two product variants on a dataset of five million users. The observed difference in means is 0.4 seconds. A stakeholder asks whether this finding is practically important. What is the most accurate statistical reasoning?
AThe result is automatically practically important, because a p-value as low as 0.001 indicates a large and meaningful effect on session duration.
BThe result cannot be trusted at all, because a dataset of five million users will produce a p-value below 0.05 regardless of the true effect.
CStatistical significance with a very large sample can reflect a tiny effect, so the practical importance depends on the effect size and context, not on the p-value alone.check_circle Correct
DPractical importance is confirmed by recomputing the p-value at a stricter significance level of 0.01, which the result already satisfies.
Statistical significance is not practical significance; effect size and context determine real-world importance. With very large samples the standard error shrinks, so trivial differences become statistically significant. Judging whether a 0.4 second difference matters requires the effect size relative to the domain, which the p-value alone cannot supply.
Why A is wrong: Tempting because a small p-value feels decisive, but the p-value measures evidence against the null, not the magnitude of the effect, so it cannot establish practical importance.
Why B is wrong: Tempting because large samples often yield small p-values, but it overstates the case: large samples increase power, they do not force significance when no effect exists.
Why C is correct: Correct: very large samples make even negligible differences statistically significant, so a low p-value does not establish practical relevance, and the 0.4 second effect must be judged against the use case.
Why D is wrong: Tempting because tightening alpha sounds rigorous, but changing the threshold still only addresses significance and says nothing about whether a 0.4 second difference matters in practice.
Exam traps in Descriptive Analysis and Visualization
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-ADS bank for this domain.
A one-way ANOVA, because it compares means across three or more groups simultaneously
errorWhy it is wrong: ANOVA tests for differences in a continuous numeric mean across groups. Both variables here are categorical counts, so comparing means is not the relevant operation and ANOVA's assumptions about normally distributed continuous outcomes do not apply.
A chi-squared test of independence between 'payment_method' and a second categorical column.
errorWhy it is wrong: Tempting because both chi-squared tests use category counts, but it is wrong: the test of independence needs two categorical variables in a contingency table, whereas this involves a single variable compared against fixed expected proportions.
The result is statistically significant at the 0.05 level because the interval is wide enough to include a 9-millisecond effect.
errorWhy it is wrong: Tempting because a 9-millisecond upper bound sounds like a real effect, but it is wrong: significance at the 0.05 level corresponds to the 95 percent interval excluding zero, and this interval contains zero, so the result is not significant.