Mental Models Daily
Explore
Menu

No. 249 · Evidence · Statistics and Probability

P-Values

A small p-value says a result would be surprising by chance, not that an idea is true.

A p-value measures how surprising your results would be if nothing real were going on. A small p-value counts as evidence against the no-effect assumption, but it does not tell you the probability an idea is true or how big an effect is.

To use it, ask how often chance alone could produce what you see. Look at the size of the effect, the sample behind it and whether others have replicated it. Be especially wary of impressive results in small or accidental slices of data.

In the Almanac

“Arbuthnot’s christenings”, 1710 · “AstraZeneca”, 2020

Both stories in full, three ways to use it and where it misleads.

Get the Almanac

Listen · Watch

P-values

Questions

What is P-Values?

A p-value measures how surprising your results would be if nothing real were going on. A small p-value counts as evidence against the no-effect assumption, but it does not tell you the probability an idea is true or how big an effect is.

How do you use P-Values?

To use it, ask how often chance alone could produce what you see. Look at the size of the effect, the sample behind it and whether others have replicated it. Be especially wary of impressive results in small or accidental slices of data.

When does P-Values matter?

P-Values is one of the 68 mental models for evidence, the situations where you ask: Is this really true? Within that, it belongs to “Testing a claim”, alongside False Negative, False Positive and Falsification.

What is an example of P-Values?

Two stories show it at work: Arbuthnot’s christenings (1710) and AstraZeneca (2020). Both are told in full in The Mental Models Daily Almanac, which has two stories for each of its 365 models.

Which mental models are related to P-Values?

Statistical Significance, Data Dredging and Replication Crisis. P-Values itself comes from statistics and probability.