Backtesting Without Fooling Yourself
Seven ways a backtest lies, and how to check for each.
Exeisy Intelligence DeskPublished 13 August 2026 · Updated 21 August 20268 min read
Key takeaways
- Look-ahead bias and unrealistic cost modelling invalidate most amateur backtests.
- Every additional tuned parameter increases the risk of describing noise.
- Reserve a held-out dataset and use it exactly once.
- Judge results by drawdown and stability across regimes, not total return.
- Forward testing under live costs is the only real confirmation.
A backtest answers one question: how would this rule have performed on this data? Whether that says anything about the future depends entirely on how carefully the test was constructed.
Seven common contaminations
1. Look-ahead bias
Using information not available at the moment of decision. The most common form is evaluating a signal on a candle's close and assuming entry at that same close.
2. Survivorship bias
Testing on instruments that still exist today, silently excluding those that were delisted or collapsed.
3. Ignoring costs
Spread, commission, swap and slippage. A strategy with a small average edge frequently becomes unprofitable once realistic costs are applied.
4. Overfitting
Adding parameters until historical performance is excellent. Each additional tuned parameter increases the chance the result describes noise.
5. Insufficient sample
Thirty trades cannot distinguish skill from luck. Results should span multiple market regimes, including at least one significant stress period.
6. Data quality
Missing candles, bad ticks and incorrect timestamps produce phantom results — often spectacular ones.
7. Optimising on the test set
Tuning on data, then reporting performance on the same data. The result is a description of the past with no predictive claim.
A defensible process
| Stage | Data used | Purpose |
|---|---|---|
| Development | In-sample (~60%) | Build and tune the rule |
| Validation | Out-of-sample (~20%) | Check it generalises |
| Final test | Held-out (~20%), used once | Honest estimate |
| Forward test | Live, small size | Confirm under real costs |
The held-out set is used once. If it fails and the strategy is revised, that set is contaminated and must be replaced.
A backtest cannot prove a strategy works. It can only fail to disprove it.
Metrics worth reading
Total return is the least informative number available. Prefer maximum drawdown, drawdown duration, return relative to drawdown, trade count, win rate together with average win/loss, and performance stability across sub-periods. A strategy that earned everything in one favourable quarter is a single observation, not an edge.
Exeisy's strategy laboratory is built around this separation between development and honest evaluation. See Technology for the architecture.
- #backtesting
- #strategy development
- #overfitting
- #validation
Disclaimer: Content provided for educational and informational purposes. Trading financial markets involves substantial risk and may result in the loss of capital. AI-assisted analysis may contain errors and should be independently evaluated.
