The Backtest Trap · Lesson 3 of 5
Twenty Trades Prove Nothing
The most common backtest error isn't a bias at all. It's believing a number that hasn't earned belief yet.
A trader tests a setup, gets 14 winners from 20 trades — 70%! — and concludes they have found something. Flip a fair coin twenty times, repeatedly, and you will regularly see fourteen heads. At twenty trials, a 70% observed rate is entirely consistent with a true 50% coin. The observation is not evidence of edge; it is evidence of twenty.
Sampling error shrinks slowly — with the square root of the trade count, which is brutal news for impatient people:
| Trades | True 50% system can plausibly show |
|---|---|
| 20 | anywhere from ~30% to ~70% |
| 50 | ~36% to ~64% |
| 100 | ~40% to ~60% |
| 500 | ~46% to ~54% |
Read the first row again: at twenty trades, a coin and a decent edge are indistinguishable. Meaningful confidence that a claimed 60% system beats coin-flipping needs a few hundred trades — and pip totals are even less trustworthy than win rates at small counts, because one outlier trade can dominate the entire sum.
Now the multiplier that makes it worse: combine small samples with lesson two's hidden attempts. Twenty trades can fake an edge one time in a handful — so a trader iterating through variants manufactures a "70% winner" every evening, purely by shopping until variance obliges. Small sample size is the fuel; multiple attempts are the match.
Practical defences: count trades, not months ("tested for a year" at one trade a week is n=52, still wide). Distrust any result an outlier dominates — recompute without the best trade and watch what remains. And when a subgroup shines ("it works brilliantly on Tuesdays!"), remember you just ran seven weekday-tests — one was always going to shine.
Our own site wears this lesson publicly: early results were labelled "n=8 — indicative, not proof" in the analysis that produced our rules, and the live record's early weeks carry the same caveat. Publishing the sample size next to every claim is the cheapest honesty there is — and its absence in anyone else's pitch tells you what you need to know.