The Backtest Trap · Lesson 5 of 5
Pre-Registration: The Honest Protocol
Science solved this exact problem decades ago. The solution transfers to trading completely — and almost nobody uses it.
Medicine faced trading's disease before trading did: researchers ran many analyses, published the flattering ones, and "discoveries" failed to replicate. The fix was not cleverer statistics — it was a change of protocol: register the hypothesis, in writing, before the evidence that will judge it exists. Prediction, not postdiction. The same protocol fits a trading desk in five steps:
1. Write the claim down, dated, specific. "Setups of type X, entered at time Y with exits Z, will clear W% over the next N trades." Vague claims are unfalsifiable, and unfalsifiable claims are marketing.
2. Freeze the rules. From registration until the sample completes, no tweaks — however clever the mid-test idea, it goes in the notebook for the next test. A rule adjusted mid-sample resets the sample, because you have started fitting the future's data the way lesson one fitted the past's.
3. Let only future data vote. The pattern was found in old data; the old data is therefore its accomplice, not its judge. Forward performance — or at minimum, held-out data the idea never touched — is the only testimony that counts.
4. Log everything, including refusals. Every signal taken and skipped, before outcomes are known. The refused pile is half the evidence: a filter only proves its worth if what it blocked did worse than what it passed.
5. Publish the verdict you got, not the one you wanted. To a partner, a forum, or just a dated file you cannot quietly edit. The knowledge that the result will exist somewhere you can't retouch changes how honestly you run steps one to four. That is the entire mechanism.
This is not theory to us; it is our operating system. Our current entry hypothesis was found in research data, so it was pre-registered — dated, specific, rules frozen — to be judged "ONLY on cycles after this date". Refusals are scored nightly in the public ledger. The research phase's losing total stays published beneath the live record. And the verdict, either way, will appear on the same pages you are reading. Watching it happen — including, possibly, watching the hypothesis die — is the best backtesting education we can offer anyone.