Data Snooping: Is Your Best Backtest Just the Luckiest?

Short answer. If you tried many versions of a strategy and report the best one, the honest question is not “is this result unlikely by chance?” but “is it unlikely among the best of all the versions I tried?”. Halbert White built a test for exactly that. In his example a model whose p-value looked like 0.0036 on its own had a p-value of 0.20 once the 3,654 models searched were counted, which is nowhere near convincing.

What is data snooping, and why is it hard to avoid?

White starts from a definition:

Data snooping occurs when a given set of data is used more than once for purposes of inference or model selection.

White, “A Reality Check for Data Snooping”, Econometrica (2000)

His point is that in time series there is usually only one history, so reuse is nearly unavoidable.¹ Run enough specifications over one price history and some will look good with no real forecasting relation behind them. He compares it to a newsletter scam: send a free tip sheet to many people, half told the market will rise and half that it will fall, keep writing only to those who got the right call, and after a few rounds some recipients have seen a perfect record by construction.

Trading parameter searches work the same way. Moving-average lengths, stop distances, session filters and entry thresholds multiply into hundreds of variants, and the winner of that contest is selected for having been lucky on this sample. We covered the first-order version of this in why backtests mislead.

What did White’s example actually show?

He applied his test to a deliberately ordinary problem: forecasting S&P 500 daily returns one day ahead. He built 29 technical indicators of the kind used by commodity traders (momentum, local trends, relative strength, moving-average oscillators) and fitted every linear model that had a constant plus exactly three of the indicators. That is 3,654 models. The benchmark was a constant alone, which embodies the idea that returns are not forecastable. The evaluation window ran from 3 June 1991 to 31 May 1994, 758 trading days.²

The best model by directional accuracy called the sign of the next day right 54.75% of the time, against 50.79% for the benchmark. That is a gap of about four percentage points, and the two p-values tell very different stories:

White’s own comment on the first row is blunt: “This example dramatically illustrates the dangers of data mining.”² A p-value of 0.0036 would pass almost any bar a trader would set. Corrected for the search, it is an unremarkable 0.20. The paper also notes that the difference between the two numbers “gives a direct estimate of the data-mining bias”.

How does the Reality Check work?

The null hypothesis is that the best model found in the search has no predictive superiority over the benchmark. The test does not ask how good the best model looks. It asks how good the best of 3,654 would look if none had any skill, and compares your actual best against that.

In practice, for a trader, a simplified version of the idea looks like this (our description of the principle, not White’s full procedure):

1. Keep the return series of every variant you tried over the same dates, not only the winner.
2. Resample the time axis (in blocks, to keep autocorrelation) and apply the same resampled dates to all variants.
3. In each resample, record the best average result across all variants, after centring each variant so that the “no skill” hypothesis holds.
4. The p-value is the share of resamples whose best result is at least as good as the one you observed.

White used a stationary bootstrap with 500 resamples for this example.² Because the whole set is resampled together, the method respects the fact that variants of the same idea are highly correlated, which matters for the next point.

Does searching more always destroy the evidence?

No, and White says so. Candidate forecasts that are highly correlated add little new “chances”; in his figures the p-value stays flat for stretches while correlated models are added, and “consideration of even a large number of models need not lead to dramatic erosion of the Reality Check p-value.”² He also argues, heuristically and without a full proof, that a genuine relationship need not be buried by properly controlled data snooping, because the power of the test increases with the sample size. In other words, the correction is not a verdict against searching. It is the price list.

Two practical consequences follow. A narrow search around one idea (ten nearby parameter values) costs far less than ten unrelated ideas. And if the naive p-value is already large, there is nothing to correct: the corrected one can only be larger.

What should a trader do with this?

• Log every variant. The test is only as honest as the set you give it. Discarded variants count, and so do the ones you stopped running after a bad week.
• Compare against a benchmark you would have used otherwise, such as holding the asset or staying flat, not against zero.
• Report the number of variants next to any result, and read results from a search as a selection effect.
• Keep untouched data. A single pre-declared test on data that played no part in choosing the rules has no search to correct for. See also how many trades prove a strategy works for the sample-size side, and survivorship bias for what the test universe silently omits.

This is educational material, not investment advice. White’s example uses an equity index in the early 1990s with linear models; the logic of the correction does not depend on the market, but the numbers are his, not ours.