The Backtest Trap · Lesson 2 of 5
The Five Ways a Test Lies
Overfitting is only the most famous liar in the room. Four quieter ones corrupt tests that were never even tuned.
1. Look-ahead bias — using information before it existed. The subtlest and deadliest. A test that enters "at the day's open" using a signal computed from the day's close has time-travelled. So has one using revised economic data (first releases get corrected later), or today's indicator settings chosen because of how the decade turned out. Audit question: at the moment of each simulated decision, was every input already knowable? One honest "no" invalidates the test.
2. Survivorship bias — testing on the winners' history. Test a stock strategy on today's index members and you have quietly excluded every company that failed — the exact events your stops exist for. In FX the trap is subtler: strategies validated only on trending darlings, or eras chosen because "the market was normal then". The deleted disasters were the test.
3. Selection bias — publishing the best of many attempts. Test twenty ideas; one shines; show that one. It feels like research — it is a lottery with extra steps, because given twenty tries, chance alone usually produces a winner. This is the engine of every too-good-to-be-true strategy thread: you see attempt #17, never attempts one to sixteen. The same trap operates privately — your discarded variations are attempts too, and their number silently devalues the survivor. (Its formal name is the multiple-comparisons problem, and it is lesson three's core.)
4. Cost amnesia — testing in a frictionless world. Spread, commission, slippage, swap: a strategy trading twice daily pays hundreds of pips a year in machinery. Backtests default to ignoring all of it, and thin-edged fast systems — the kind naive tests love most — are precisely the kind costs erase. If the tested edge per trade is smaller than the real cost per trade, the beautiful curve inverts in production.
5. Regime dependence — one era, one verdict. A strategy tested on 2020–2024 was tested on those years' interest-rate regime, volatility and trends. It learned that world, and the next world is under no obligation to resemble it. Honest tests span regimes — or admit, out loud, that they could not.
We wrote that admission verbatim: our research fortnight was "one market regime — treat as a strong first read, not final proof". The same review also caught bias #3 in our own conclusions: a promising pattern mined from the data it was judged on. It was demoted to a hypothesis and sent to forward testing — next lesson explains why that was the only honest option.