The Backtest Trap · Lesson 4 of 5
Choose Plateaus, Not Peaks
When a parameter sweep crowns a winner, the winner is usually the luckiest cell in the spreadsheet — not the best rule.
Sooner or later every systematic trader runs a grid: stops of 15, 25, 40, none; targets of 25, 40, none; holds of 4, 8, 24 hours — every combination scored. Out comes a table with one glowing cell, and every instinct says: ship the glowing cell. This lesson is one idea, worth its own place in the course: the peak cell is where luck concentrated; the plateau is where truth lives.
Think about what a grid actually measures. Each cell's score = the rule's real merit plus that cell's share of sample luck. Merit varies smoothly — a 38-pip stop and a 42-pip stop are nearly the same rule and must have nearly the same true value. Luck varies cell by cell. So when one cell towers over its neighbours, the towering part is, by elimination, mostly luck: the merit component cannot spike between adjacent settings, only the noise can.
Hence the reading protocol:
Look for regions, not cells. A broad neighbourhood of settings that all perform decently — a plateau — indicates a rule whose merit does not depend on magic numbers. That robustness is the thing most likely to survive the future.
Interrogate the champion's neighbours. If the best cell's adjacent settings are mediocre, the champion is an artefact. Discard it especially because it looks best — its excess over the plateau is a measure of luck, not edge.
Pick from the plateau's centre, then stop. Choose a representative setting from the robust region — deliberately NOT re-optimising within it — and accept that your chosen cell will look worse than the champion on the past. It will usually look better than the champion in the future, which is the only place you trade.
Our exit grid was a textbook specimen: tight stops bad everywhere (a plateau of failure), take-profits helpful everywhere (a plateau of merit), and one glowing no-stop cell at +151 pips towering over sane neighbours. We shipped the plateau — stop 40, target 25 — and published the refusal of the peak. On the sample, our choice "cost" us the difference; that difference was never real money, only luck's ghost.