The rules can be perfectly written and the engine perfectly fair, and the test can still mislead, because the data fed into it is not what a trader would have seen at the time. Data errors rarely make a strategy look worse. They almost always make it look better, which is exactly why they survive: nobody goes looking for a bug in a result they like. This lesson covers the five traps that matter most for rule-based stock trading.
Survivorship bias
Ask a data provider for “the 500 most liquid US stocks” today and you receive a list of companies that are, by definition, still here. Companies that went bankrupt, were delisted after a long decline or were acquired between 2000 and today are missing. A test that runs today’s list back to 2000 is testing a portfolio of future winners, chosen with hindsight.
The effect is largest for long-only rules on broad lists of individual stocks, and over long periods. It is smallest for tests on index ETFs such as SPY, which hold whatever the index held at the time, so their own price already includes the companies that failed.
A complete fix requires survivorship-free data: a database that includes delisted companies and records which stocks belonged to an index on each date. Such data exists, but it is costly and its licences rarely allow it to be shown to other people. A practical partial fix is to make the benchmark share the bias. Clear Tape’s holdout verdict compares the strategy with the same stocks bought and held in equal weights: both sides then contain the same future winners, and much of the bias cancels in the comparison. Clear Tape also states on every result that only stocks listed today are tested, and marks tests on its 500-stock list as “Bias: high”, because that list is chosen by today’s liquidity.
Look-ahead bias
Look-ahead bias is any use of information that was not available when the decision was made. It is easy to introduce and hard to notice, because the code still runs and the result still looks plausible. Typical forms:
- Same-bar decisions. Deciding at the open using that day’s close, high or low. A rule that “buys at the open if the day closes above the 20-day high” cannot be traded.
- Indicators that include the current bar where they should not. Comparing today’s close with a 20-day high that already contains today.
- Lists chosen later. Using today’s index members, today’s sector labels or today’s “most liquid” ranking to define a universe in the past.
- Revised data. Using economic or company figures as they read after later revisions, rather than as first published.
- Whole-sample statistics. Normalising a signal with the average and standard deviation of the entire history, which include the future.
The defence is a timeline. For each piece of data in a rule, write down the moment it became known, and check that every decision happens after it. Signals on the close, filled at the next open, are the safest default. One strong test: delete all data after a date and rerun. If results before that date change at all, something is reading the future. Clear Tape’s holdout test is designed around exactly this property: removing every price after 2014 must leave the training result unchanged.
Splits, dividends and adjustments
When a company splits its stock 10 for 1, the share price falls by 90% overnight and the holder owns ten times as many shares; nothing of value has changed. In raw price data, though, it looks like a crash. Any rule that reads raw prices across a split will see a collapse that never happened: a stop is hit, a 52-week high becomes unreachable, a moving average lurches.
Data providers therefore publish adjusted prices. Two adjustments are common:
- Split adjustment rescales history so that splits leave no gap. Every price-based signal should use split-adjusted data.
- Dividend adjustment also lowers earlier prices by each dividend, so that the adjusted series behaves like a total-return index. It is convenient for measuring returns, but it changes past price levels: a stock that really traded at $50 in 2015 may show $41 in today’s adjusted series, and the figure keeps changing as new dividends are paid.
A sound approach keeps the two apart: compute signals on split-adjusted prices, which match the levels traders actually saw, and add dividends as cash when they are paid. That is how Clear Tape’s holdout test treats them. Whatever the method, apply it identically to the strategy and the benchmark (lesson 3).
Delisted stocks and point-in-time data
Two further gaps complete the list.
Delisted stocks. Even when price history for a delisted company is available, a test also needs to know what happened at the end: a cash takeover pays out at the deal price, while a bankruptcy may leave shares nearly worthless. A test that simply stops on the last available bar treats both the same, and it typically misses the final loss. When a rule tends to buy stocks in steep decline, this gap is especially costly, because those are the stocks most likely to disappear.
Point-in-time data. Company figures arrive weeks after a quarter ends and are sometimes restated months later. A database that stores only the latest, corrected figure, back-dated to the end of the quarter, lets a test act on numbers nobody had yet. Point-in-time data records each figure as it was known on each date.
The same applies to index membership, analyst estimates and economic releases. Clear Tape’s rules are built from price and volume only, so this trap does not arise inside its backtests, but it matters for any test you run elsewhere that uses fundamentals.
How to apply it
Entry
Before testing, write down the data source, how it is adjusted, whether it includes delisted stocks, and the moment each input becomes known.
Stop
If a result depends on a stock list chosen today, compare it with the same list bought and held, and read the comparison, not the absolute return.
Target
A test in which every input is available before each decision, prices are split-adjusted, dividends are treated the same on both sides and the remaining biases are stated with the result.
Invalidation
Results that change when later data is deleted, gaps in a price series around splits, or a universe defined with hindsight against an unbiased benchmark: fix the data before reading the result.
Common mistakes
- Testing today’s index members or most-liquid stocks back over twenty years and comparing the result with SPY.
- Using raw prices across splits.
- Mixing a dividend-adjusted strategy with a price-only benchmark.
- Treating a stock that disappears from the data as if it had been sold at its last price.
- Using restated fundamentals as if they had been known at the end of the quarter.
Summary
Data traps almost always flatter a test. Survivorship bias comes from testing only companies that still exist; a same-stock benchmark cancels much of it, and it should be stated with every result. Look-ahead bias comes from using information before it was available; a written timeline and the “delete the future” check catch most of it. Signals belong on split-adjusted prices, with dividends counted the same way for strategy and benchmark. Delisted companies and restated fundamentals need data built for the purpose. The next lesson moves from the test to the market: how to start trading a rule that has passed, and how to tell when it has stopped working.
For educational purposes only. Not investment advice.