A rule usually has numbers in it: a 200-day average, a 20-day breakout, a stop at 2 × ATR. A parameter sweep runs the same rule many times with different numbers to see how much the result depends on them. Used well, a sweep is one of the best robustness checks available. Used badly, as a search for the single best setting, it is one of the fastest routes to an overfit rule. The difference lies entirely in how the results are read.
What a sweep is
A sweep varies one or two parameters across a grid of values. A one-parameter sweep might test moving-average lengths of 50, 75, 100 and so on up to 250 days. A two-parameter sweep might cross five entry lookbacks with six exit lookbacks, giving 30 combinations. Each combination is a variant, and each is run over the same training period with the same costs.
Clear Tape limits sweeps to two parameters, two to eight values per parameter and at most 30 combinations, on up to 50 stocks, and the grid must include the settings you started with. Those limits are partly about computing time, but they also cap how many variants you can try, which is itself a protection.
Multiple testing: the best of many is flattered
Every backtest result is part edge and part luck. With one test you cannot tell the two apart, but at least the luck is as likely to hurt as to help. Run thirty tests and keep the best, and you have selected the one where luck helped most. The more variants you try, the more luck the winner contains.
A simulation makes the point. Thirty strategies were generated whose daily returns are pure noise: their true edge is exactly zero. Over 15 years, the length of Clear Tape’s training period, their Sharpe ratios still spread out widely:
A Sharpe ratio of 0.54 would look respectable in any backtest report. Here it is noise. A rough rule of thumb says the same: if each test has a 5% chance of looking significant by luck alone, the chance that at least one of 20 independent tests does so is 1 − 0.95^20, about 64%; for 30 tests it is about 79%. Settings of the same rule are not fully independent, which softens the effect, but does not remove it.
Researchers have taken this seriously. Campbell Harvey, Yan Liu and Heqing Zhu argued that, given how many factors have been tested on the same data, many published stock-market factors are likely to be false discoveries and the bar for significance should be raised. David Bailey and Marcos López de Prado proposed the deflated Sharpe ratio, which discounts a result for the number of trials behind it. Clear Tape does not compute that adjustment; instead it shows how many variants were tested next to every sweep, and judges a sweep by its shape rather than its peak.
Plateaus and isolated peaks
A real effect is rarely tied to one exact number. If a 150-day average works because markets trend over months, then 125 and 175 days should work nearly as well. Luck, by contrast, is local: it lands on one setting and spares the neighbours. So the useful question is not “which cell is best?” but “does the best cell sit on a plateau or on an isolated peak?”
Clear Tape applies the plateau rule to the training period only:
- Each cell’s score is its training Sharpe ratio minus the benchmark’s training Sharpe ratio. Cells with fewer than 20 closed training trades are left out.
- The best cell is the valid cell with the highest score. If that score is zero or below, the sweep reads No setting beat the benchmark.
- Its neighbours are the valid cells next to it on the grid, diagonals included; in a one-parameter sweep, the values either side. With fewer than two neighbours the sweep reads Too few neighbouring settings; add values around the best one.
- It is a Plateau when the median neighbour score is at least half the best score and at least two thirds of the neighbours score above zero. Otherwise it is an Isolated peak, which is likely luck.
In the left panel above, the best length (150 days, +0.17) has neighbours at +0.14 and +0.15: a median of +0.145, more than half of +0.17, and both positive, so it is a plateau. On the right, the best length (125 days, +0.21) has neighbours at −0.03 and −0.05: the median is negative, so it is an isolated peak, however good +0.21 looks.
Reading a heat map
A two-parameter sweep is shown as a heat map: one axis per parameter, each cell coloured by its training score, green above the benchmark and red below.
Read it in this order:
- The overall shape. A broad region of similar colour is evidence that the idea, not the exact number, carries the result. A checkerboard or a single bright cell is evidence of noise.
- Your own settings. They are outlined. If they sit inside a plateau, good. If they sit on an isolated peak, or the plateau is somewhere else entirely, your settings were probably chosen with hindsight.
- The blank cells. Settings with too few trades are not scored at all, rather than being allowed to win on a handful of lucky trades.
- The number of variants tested. Thirty cells means thirty chances for luck.
Notice what the heat map does not show: the holdout period. Clear Tape colours the grid with training results only, and gives a full holdout verdict for one cell, your own settings. If the grid showed holdout results too, it would be tempting to choose the cell that did best in the holdout, and the holdout would stop being a fair test. If the sweep suggests that different settings would be better, you can make that change, but the result is a new study whose holdout has, in effect, already been glimpsed.
How to apply it
Entry
Choose your settings before the sweep, for a reason you can state. Build the grid around them, with values on both sides.
Stop
Stop when the shape reads as an isolated peak or no setting beats the benchmark. Do not widen the grid in search of a better corner.
Target
A plateau that contains your own settings, built from cells with enough trades, followed by a holdout verdict on those settings.
Invalidation
If you change your settings to the best cell after seeing the sweep, record the study as tuned and treat any holdout result with extra caution.
Common mistakes
- Picking the brightest cell and calling its number the edge.
- Sweeping many parameters at once until something works.
- Building the grid so that your setting sits on its edge, with no neighbours on one side.
- Counting cells with very few trades as evidence.
- Checking the holdout for several cells and keeping the best.
Summary
A sweep shows how sensitive a rule is to its numbers. The best of many variants is flattered by luck: even when thirty strategies have no edge, the best of them shows a fifteen-year Sharpe ratio that passes for skill. Judge a sweep by its shape. Clear Tape scores each cell by training Sharpe ratio above the benchmark, ignores cells with fewer than 20 trades, and calls the best cell a plateau when its neighbours’ median is at least half its score and two thirds of them are positive, an isolated peak otherwise. The holdout verdict is given for your own settings only, so the holdout stays a fair test. The next lesson turns to the data itself, and the traps hidden in it.
For educational purposes only. Not investment advice.