Stories. Questions. Different perspectives.
Browse WAURE
← Advice & Questions
Education · Advice & Questions ·

Does a statistically significant result mean the improvement is important in practice?

Education category photographEducation

A made-up example in our statistics class reports a difference of 0.1 points on a 100-point test and labels it statistically significant. I expanded that into “a major educational improvement” in my summary.

The American Statistical Association's statement says statistical significance does not measure the size of an effect or the importance of a result. That makes me doubt the wording I chose.

What else should our summary examine? I do not want to dismiss a small difference automatically, but the significance label alone does not seem to answer whether changing the teaching approach would be worthwhile.

Discussion

5 community replies

Report the estimated difference in its actual units before adding an importance judgment. Here the example gives 0.1 points on a 100-point test; the significance label does not turn that into a larger score change.

Ask what size of difference would matter for the particular educational purpose and why. There is no threshold of practical importance supplied merely by calling the result statistically significant.

A small effect is not automatically worthless. Its consequences could depend on context, burden and other outcomes. Those are questions to examine, rather than reasons to replace one automatic verdict with another.

Look for information about uncertainty and the study design as well as the estimate. Ask the tutor what the example provides and what it leaves unspecified before recommending an intervention.

Make those limits visible in the summary. You can explain the reported result and identify what a practical recommendation would still need, without presenting the p-value as either a complete endorsement or a complete dismissal.

Join the discussion

More in Education