Backtesting Without Fooling Yourself

4.1 The perfect strategy

Node found a strategy online with a remarkable backtest: 300% a year, a maximum drawdown of only 8%, and a smooth equity curve. It used eleven indicators with very specific settings: a 17-period RSI, a 43-period moving average, a 2.3 standard deviation Bollinger Band.

Node tried it live. Within two months, it had lost 20%.

"The backtest was perfect," said Node.

"That's what made it suspicious," said Hash. "Very specific settings that produce near-perfect historical results usually mean the strategy was fitted to the past, not to anything that repeats. It's the most common way backtests mislead."

4.2 Overfitting

Overfitting happens when a strategy is tuned so closely to historical data that it captures random noise rather than a genuine pattern. It looks excellent in the past and fails in the future.

A simulation shows how easily this happens. Imagine testing 1,000 strategies that have no edge at all: each trade is a pure coin flip, winning or losing 1R, over 250 trades.

Result across 1,000 zero-edge strategies (simulated)	Value
Best strategy's result	+52R
Strategies showing +20R or more	111
Worst strategy's result	−60R

The best "strategy" made 52 units of risk over 250 trades, which looks like a solid edge. It was pure luck. When the ten best performers were tested again on new random data, their results ranged from −24R to +48R, averaging only about +7R, far below the gains they'd shown before and just as likely to be negative as positive.

"So if I test enough ideas, some will look brilliant by chance," said Mint.

"And if you only report the best one, you'll believe it," said Hash. "This is called data snooping: testing many variations on the same data until something works."

4.3 Signs of overfitting

Warning sign	Why it's suspicious
Many indicators and conditions	More ways to fit noise
Oddly specific parameters, like 17 and 43	Likely chosen because they happened to work best
Performance collapses with small parameter changes	A real edge shouldn't depend on exact settings
Near-perfect equity curve	Real strategies have losing periods
Works on one asset or period only	Pattern may be specific to that history
Many variations tested, only the best reported	Data snooping

Robustness test. Change each parameter slightly. If a 50-day moving average works, do 40 and 60 also work reasonably? Robust strategies degrade gradually. Overfitted strategies fall apart.

Moving average length	Robust strategy result (illustrative)	Overfitted strategy result (illustrative)
40	+0.7R per trade	−0.2R
43	+0.75R	+1.9R
50	+0.8R	−0.1R
60	+0.7R	−0.3R

4.4 Out-of-sample testing

The best defence against overfitting is testing on data the strategy was never designed on.

Data split	Use
In-sample	Develop and adjust the strategy
Out-of-sample	Test once, at the end, without further changes

If a strategy performs well in-sample but poorly out-of-sample, it was probably overfitted. The out-of-sample data should be touched as rarely as possible: every time you adjust the strategy after looking at it, it stops being truly out-of-sample.

Walk-forward testing repeats this process: develop on one period, test on the next, then roll forward. It simulates how the strategy would have been built and used in real time.

Chapter 4 · Overfitting and out-of-sample

A strategy is built and optimised on an in-sample period, where its equity curve looks excellent. Tested once on a reserved out-of-sample period it never saw, an overfit strategy's performance collapses, while a robust one holds up. The gap between the two reveals curve-fitting.

4.5 Look-ahead bias

Look-ahead bias means using information in a backtest that wouldn't have been available at the time.

Example	Problem
Entering at a day's open based on that day's close	You can't know the close at the open
Using a candle's high to trigger an entry, then filling at the low	Unrealistic order of events within a candle
Using revised or later-adjusted data	Original data may have been different
Using a token's later category or fundamentals	Wasn't known at the time

Look-ahead bias is often a simple coding mistake, and it can make a worthless strategy look excellent.

4.6 Survivorship bias

Testing a strategy on today's top 50 tokens over the past five years ignores tokens that were in the top 50 five years ago but later collapsed or disappeared. Those tokens are exactly the ones a real strategy would have bought and lost money on (Module 12, Chapter 11).

To reduce survivorship bias, use historical lists of what existed and qualified at each point in time, including tokens that later failed.

4.7 Costs and realism

Cost or friction	Effect if ignored
Trading fees	Small per trade, large for frequent strategies (Module 7, Chapter 11)
Slippage	Real fills worse than backtest prices, especially in thin markets
Funding	Perp strategies pay or receive funding over time (Module 7, Chapter 7)
Liquidity limits	Can't trade unlimited size at the backtest price
Downtime and outages	Exchanges halt during the most volatile moments
Taxes	Frequent trading may create tax costs (Module 14, Chapter 2)

Example: a frequent trading strategy.

	Per trade
Expectancy before costs	+0.15R
Fees and slippage	−0.10R
Expectancy after costs	+0.05R

Two-thirds of the edge disappears to costs. With slightly higher costs, it would lose money.

4.8 A backtest honesty checklist

1. Did I start with a hypothesis before testing? (Chapter 2)
2. How many variations did I test, and am I reporting only the best?
3. Do nearby parameters also work?
4. Did it work out-of-sample, on data I didn't use to design it?
5. Did it work on other assets or periods?
6. Did I avoid look-ahead and survivorship bias?
7. Did I include realistic fees, slippage, and funding?
8. Are there enough trades to trust the results?
9. Does the equity curve look realistic, with losing periods?

4.9 Try it yourself

Take the backtest from Chapter 3. Change the moving average length to 40, 45, 55, and 60 days, and record results for each. Then split your data into two halves: develop on the first half, and test once on the second. Finally, double your cost assumption and see how much the results change.

Key takeaways from this chapter

1. Overfitting tunes strategies to past noise; they look excellent historically and fail in the future.
2. Testing many zero-edge variations will produce impressive-looking winners purely by chance.
3. Warning signs include many indicators, oddly specific parameters, fragile results, and near-perfect curves.
4. Out-of-sample and walk-forward testing are the best defences against overfitting.
5. Look-ahead bias uses information not available at the time; survivorship bias ignores failed assets.
6. Realistic costs can erase much of an edge, especially for frequent trading.
7. Use an honesty checklist before trusting any backtest.