Give a computer enough freedom and it will find a rule that made money on any stretch of history. Add a filter for each losing year, pick the moving-average length that happened to work best, exclude the month that hurt: each step improves the backtest and each step teaches the rule a little more about the past and a little less about markets. That process is called overfitting, and it is the most common reason a beautiful backtest turns into a disappointing account. The standard defence is simple: keep part of history out of sight while the rule is designed, then test on it once.
What overfitting looks like
Price history is a mixture of repeatable behaviour and noise: one-off events, random sequences, coincidences of timing. A rule with few moving parts can only capture broad behaviour. A rule with many conditions and finely tuned numbers can also capture the noise, and in a backtest the noise looks exactly like profit.
Warning signs of an overfit rule:
- Many conditions, each of which was added after looking at a losing period.
- Oddly specific numbers: a 37-day average, an RSI threshold of 23, a stop at 6.4%.
- A result that collapses when a setting moves slightly (lesson 5).
- Most of the profit coming from a handful of trades or from one year.
- No plain-language reason why the rule should work.
In-sample and out-of-sample
Data used while designing, choosing or tuning a rule is in-sample. Every decision made while looking at it, including the decision to keep the rule at all, has used that data, so the in-sample result is biased upwards. Data the rule has never been shaped by is out-of-sample, and only there does a test measure something close to how the rule might behave on new prices.
The simplest arrangement splits history in two. The training period is where the rule is judged and, if you wish, refined. The holdout period is set aside and run once, with the rule unchanged. If the rule did well in training only because it fitted noise, that advantage has no reason to reappear in the holdout.
The holdout keeps its value only while it stays unseen. If you run the holdout, dislike the result, adjust the rule and run it again, the holdout has become training data, and the new result is in-sample in disguise. One look is the rule; a second look after a change is a new study whose holdout result deserves much less trust.
Why a fixed split
If the split date can move, it becomes one more parameter to tune: try 2010, then 2012, then 2015, and stop at whichever gives the nicest holdout. To remove that temptation, Clear Tape uses one split for every test:
- Training: 1 January 2000 to 31 December 2014. Fifteen years including the 2000–02 bear market, the 2007–09 financial crisis and long advances in between.
- Holdout: 1 January 2015 to the latest close. More than a decade including the 2018 sell-off, the 2020 crash and the 2022 bear market.
- Warm-up from 1998, so that a 200-day average already has a value on the first day of training.
Because the split is the same for everyone and every rule, verdicts can be compared across studies. The cost is that stocks listed after 2015 have no training period, so a test on them returns Not enough data rather than a verdict built on a few years. The honest remedy is to test the idea on an ETF or on stocks with a longer history, not to relax the standard.
A trade that is open on 31 December 2014 is valued at that day’s close for the training result and does not count in the holdout, which includes only trades entered from 2015 onward. Indicators are computed on the whole series using only past data, so nothing from the holdout leaks into training.
The four verdicts
Each holdout test in Clear Tape ends with one of four verdicts. The strategy is compared with its benchmark (the same stocks bought and held in equal weights, total return) on two measures, CAGR and the Sharpe ratio, in each of the two periods: four comparisons in all. “Beat” means strictly higher after rounding to the displayed precision (CAGR in percent to one decimal place, the Sharpe ratio to two decimal places); a tie counts as not beaten.
| Verdict | Condition |
|---|---|
| Not enough data | In either period: fewer than 3 years of benchmark prices, fewer than 20 closed trades across all stocks, data for fewer than half the stocks tested, or no risk-free rate or benchmark. Checked first. |
| No edge | No win on either measure in the training period, or no win on either measure in the holdout period. |
| Passed holdout | Wins on both CAGR and Sharpe ratio in both periods: four out of four. |
| Trade-off | Everything else: at least one win in each period, but not all four. Typically higher return with deeper swings, or a smoother ride with less return. |
The rule is strict on purpose. A rule that fails in training but happens to win in the holdout is still No edge: the rule was formed on the training data, so failing there is the more informative result. Five hypothetical studies show how the order works:
| Hypothetical study | Training: CAGR, Sharpe (benchmark) | Holdout: CAGR, Sharpe (benchmark) | Verdict |
|---|---|---|---|
| A | 8.4% (7.1%), 0.52 (0.38) | 11.9% (11.2%), 0.81 (0.74) | Passed holdout |
| B | 6.2% (7.1%), 0.49 (0.38) | 9.8% (11.2%), 0.79 (0.74) | Trade-off |
| C | 9.0% (7.1%), 0.55 (0.38) | 10.6% (11.2%), 0.70 (0.74) | No edge |
| D, 18 training trades | 10.3% (7.1%), 0.66 (0.38) | 12.5% (11.2%), 0.90 (0.74) | Not enough data |
| E | 7.4% (7.1%), 0.40 (0.38) | 11.24% → 11.2% (11.2%), 0.76 (0.74) | Trade-off |
Study C beat its benchmark comfortably in training and still ends as No edge: it lost both comparisons once it met new data, the classic signature of overfitting. Study D has the most attractive numbers of all and receives no verdict, because 18 trades are too few to separate skill from luck. Study E ties on holdout CAGR after rounding, so that comparison counts as lost.
Why “No edge” is a useful result
Most honest tests tend to end in No edge or Trade-off. That is not a sign that the method is broken; it reflects how hard it is to beat simply holding the same stocks once costs, dividends and cash interest are counted on both sides. A No edge verdict tells you something valuable: this rule, on these stocks, did not earn its complexity. It saves you the money and months you would have spent learning the same thing live.
Record No edge results as carefully as the good ones. A research journal that keeps only the winners hides how many ideas were tried, which makes the winners look more convincing than they are (lesson 5). Clear Tape lets you save every study whatever its verdict, and shows all four verdicts in the same size and place so that none is played down.
How to apply it
Entry
Design and refine the rule using the training period only. Write the final version down before running the holdout.
Stop
Run the holdout once. If you change the rule afterwards, record it as a new study and treat its holdout result with less confidence.
Target
Look for Passed holdout, or a Trade-off whose trade-off you would genuinely accept, such as lower return for a much smaller drawdown, supported by a reasonable number of trades in both periods.
Invalidation
No edge ends the study. Not enough data means the question cannot be answered on this list of stocks; test the idea on instruments with longer histories instead.
Common mistakes
- Peeking at the holdout while designing the rule, then calling the result out-of-sample.
- Moving the split date until the holdout looks good.
- Re-running the holdout after each change until the verdict improves.
- Reading a Trade-off as a pass because the better half of it is the half you care about today.
- Discarding No edge studies, so the journal shows only successes.
Summary
Overfitting is fitting rules to the noise of the past; it improves in-sample results and harms out-of-sample ones. A holdout period kept unseen and run once is the simplest defence, and a fixed split stops the split itself from being tuned. Clear Tape trains on 2000–2014 and holds out 2015 onward, compares CAGR and Sharpe ratio with an equal-weight, total-return benchmark in both periods, and reports Passed holdout, Trade-off, No edge or Not enough data in that strict order. No edge is a common, useful answer. The next lesson looks at another route to overfitting: trying many parameter settings and keeping the best.
For educational purposes only. Not investment advice.