Some of the most common analytical mistakes come from statistical traps that catch experienced people too.
Simpson's Paradox
A trend in combined data can reverse within every subgroup. A treatment can look worse overall yet better for both mild and severe cases, if severe cases were more likely to get it. Always check important results within relevant segments.
Regression to the Mean
Extreme results tend to be followed by more ordinary ones. The worst-performing stores this month will often improve next month without any intervention — so "improvements" after targeting outliers can be illusory. Use control groups.
Survivorship Bias
Looking only at cases that survived — successful companies, planes that returned — hides the failures that would change the conclusion.
Multiple Comparisons and p-Hacking
Testing many hypotheses, metrics or segments guarantees some "significant" results by chance. Pre-register the main question, adjust for multiple tests and treat surprising subgroup findings as hypotheses to confirm.
Base-Rate Neglect
Ignoring how common something is. A highly accurate test for a rare condition still produces mostly false positives.
Small Samples
Rates from small groups swing wildly. The best and worst performers are often simply the smallest.
Ecological Fallacy
Drawing conclusions about individuals from group-level data.
Defence
Plot the data, segment results, compare with baselines and controls, report uncertainty, and ask a colleague to challenge your conclusions.