Overfitting, also called curve fitting, happens when a trading strategy's rules are tuned so closely to past prices that they describe that history's random noise rather than a behavior that repeats. An overfit strategy looks excellent on the data it was built on and disappoints on data it has not seen, which is why an impressive backtest alone proves little.
How it works
Price history mixes repeatable behavior, such as trends that persist for months, with noise: one-off news, coincidences of timing, random runs of wins and losses. A rule with few conditions can only capture broad behavior. Every extra condition, filter or tuned number gives the rule more freedom to bend around individual past trades, and in a backtest a fit to noise looks exactly like profit.
The freedom grows quickly. A rule with five parameters, each tried at six values, has 7,776 combinations; one of them will look strong on almost any data.
Warning signs
- Results that look too smooth: a very high win rate, a tiny drawdown or a profit factor far above what similar rules show.
- Oddly precise settings, such as a 37-day average or a 6.4% stop, with no reason other than "it tested best".
- A sharp parameter peak: the chosen value works, but its neighbors do poorly.
- Few trades: thirty trades spread over many conditions leave each condition supported by a handful of examples.
- Rules added after looking: a filter written to remove last year's losing trades will, by construction, improve last year.
In-sample vs out-of-sample testing
The in-sample period is the data used to design and tune the rules. The out-of-sample period is data kept aside and not looked at until the rules are final. Only the out-of-sample result says anything about how the rules handle conditions they were not fitted to. The simplest version is a two-period test: split history at a fixed date, build on the earlier part (the rule period), then run the finished rules once on the later part (the test period). If the test-period result is then used to adjust the rules, it becomes in-sample data and the protection is gone. The Academy lesson on overfitting and the test period covers this split in detail.
Walk-forward analysis
Walk-forward analysis repeats this split many times. Tune the rules on an in-sample window, test them on the period that follows, then move both windows forward and repeat. For example, with data from 2010 to 2024, a five-year in-sample window and a one-year test window, the rules are tuned on 2010–2014 and tested on 2015, tuned on 2011–2015 and tested on 2016, and so on: ten separate one-year tests from 2015 to 2024. Joined together, those test years form an out-of-sample record. What is being tested is the whole procedure, including how the parameters are chosen, not one fixed setting.
Plateau vs peak
A real effect rarely depends on one exact number. If a 50-day breakout works because trends persist, 40 and 60 days should work nearly as well: the results form a plateau. Luck lands on one setting and spares its neighbors, producing an isolated peak. A parameter sweep is therefore read for its shape, not for its best cell; the lesson on parameter sweeps shows how.
Multiple testing and data snooping
Each backtest result is part edge and part luck. Run one test and luck is as likely to hurt as to help; run fifty and keep the best, and the winner is the one luck helped most. As an illustration, if 20 unrelated rules with no edge at all are each judged at a 5% significance threshold, the chance that at least one passes by luck alone is about 64%. Data snooping is the same effect spread over time: reusing one dataset for idea after idea until something works.
Worked example (illustrative numbers)
| Made-up rule version | Parameters | In-sample trades, expectancy | Out-of-sample trades, expectancy |
|---|---|---|---|
| Simple: breakout, stop, exit | 3 | 150 trades, +0.15R | 140 trades, +0.10R |
| Tuned: plus four filters added after studying losers | 7 | 40 trades, +0.50R | 35 trades, −0.05R |
These numbers are made up to show the pattern. The filters more than tripled the in-sample expectancy while cutting the trade count, and the out-of-sample run, made once, shows the gain was fitted to the past. The simple version weakened out of sample, as most rules do, but kept its shape.
How to avoid overfitting
- Write the rules and the reason for them before testing, as part of a trading system.
- Use few parameters and round values, and prefer rules that work across a range of settings.
- Demand enough trades: a few hundred across different market conditions say far more than a few dozen.
- Include realistic costs; a thin edge often disappears once commissions and slippage are charged.
- Keep out-of-sample data untouched and use it once. Count every variant you tried, including the ones you discarded.
- Check the data itself for survivorship and look-ahead problems, covered in the lesson on data traps.
Limitations and common mistakes
- Passing out of sample is not proof. It lowers the odds of a pure fit, but markets change and live results usually trail tests; see from backtest to live trading.
- Peeking repeatedly at the test period, then adjusting, quietly turns it into in-sample data.
- Judging by one statistic. Optimizing for the highest profit factor or win rate invites curve fitting; read expectancy, trade count and drawdown together.
In Clear Tape
In Systems › Backtest, the two-period test judges a system on the rule years (2000–2014), then runs it unchanged in the test years (2015 to now) against holding the same stocks, and ends in one of four verdicts: Beat buy-and-hold in both, Beat buy-and-hold in part, Did not beat buy-and-hold or Not enough data. Parameter sweeps of up to two parameters run on the rule years only, so the test years stay unseen, and every result discloses its costs and survivorship bias. Rules can start from the System library of published rules, each with its source. A free member can backtest one library rule a day; Tools members run 20 backtests and 3 sweeps a day, Pro members 50 and 10 (plans).
A test only covers the rules as written. Once a system is traded, the free trade journal records each trade under its system, shows statistics broken down by system or setup, and the AI trade score checks whether each trade followed that system's rules, so live results can be compared with the tested rules rather than with a version improvised along the way.
For educational purposes only. Not investment advice.