No. 253 · Evidence · Statistics and Probability
Statistical Significance
A significant result is one that chance alone would rarely produce; it need not be big.
In the Almanac
“Student’s t-test”, 1908 · “Remdesivir”, 2020
Both stories in full, three ways to use it and where it misleads.
Get the AlmanacListen · Watch
Questions
What is Statistical Significance?
Statistical Significance asks whether a result would be unlikely if there were no real effect. William Gosset, a Guinness brewer writing as Student in 1908, invented the t-test for small samples. In 2020, remdesivir significantly sped recovery, but its effect on deaths wasn't significant.
How do you use Statistical Significance?
Before declaring a winner, ask whether you have enough data, how big the effect is, and whether it replicates. Significant doesn't mean important, and not significant doesn't prove there's no effect.
When does Statistical Significance matter?
Statistical Significance is one of the 68 mental models for evidence, the situations where you ask: Is this really true? Within that, it belongs to “Testing a claim”, alongside False Negative, False Positive and Falsification.
What is an example of Statistical Significance?
Two stories show it at work: Student’s t-test (1908) and Remdesivir (2020). Both are told in full in The Mental Models Daily Almanac, which has two stories for each of its 365 models.
Which mental models are related to Statistical Significance?
P-Values, Error Bars and Monte Carlo Fallacy. Statistical Significance itself comes from statistics and probability.