/ Check 9 · Selection

Is your optimized backtest overfit?

Start with a number most reports leave out: how many settings were tried before this one was picked. The best of forty is held to a higher bar than one test run once, and the usual 95% interval does not know the other thirty-nine existed.

The short answer

Every optimizer has a best row, whether or not anything in the sweep works. A result picked from a sweep has to clear a bar that rises with the number of settings tried. Each result also has to look like its neighbours. If you do not know how many settings were tried, you do not know what the reported confidence means.

What counts as a variant

  • Every parameter combination run. Eight lookbacks times five stop distances is forty variants, including the thirty-nine nobody looked at twice.
  • Every filter added after seeing results. Dropping the worst weekday, adding a trend filter, skipping a bad month: each one is another draw.
  • Every window edge moved. Starting the test a year later because the first year looked bad is a selection, even if only one run was saved.
  • Every strategy abandoned on the way. Ideas that were tested and dropped still count, because they are why this one is the one being shown.

A worked example: the best of forty

A made-up strategy is swept over 8 lookbacks and 5 stop distances, forty variants, each producing 200 trades with wins and losses about the same size. Break-even at that payoff is 50%. The best variant wins 120 of 200, a 60% win rate.

Taken on its own, that looks like a result. The 95% interval on 120 of 200 is 53.1% to 66.5%, entirely above break-even. Now give it the history it actually has.

  • Noise clears the usual bar most of the time. If none of the forty variants had any edge, the chance that at least one would still pass a two-sided 5% test is 1 − 0.9540 = 87%.
  • Noise hands you a respectable best row. With 200 trades at a true 50%, the expected best of forty unrelated coin-flip variants is about 57.6%, roughly 115 wins. The chance that at least one of the forty reaches 120 or more is about 11%, close to one sweep in nine.
  • The bar moves. Spread a 5% error rate across forty tests (Bonferroni) and the winner needs p below 0.05 ÷ 40 = 0.00125, which is a z of about 3.23 instead of 1.96. At that bar the interval on 120 of 200 is 48.6% to 70.4%. Its lower end is back below break-even.

So the same 120 of 200 passes as a single test and fails as the best of forty. Nothing about the trades changed. Only the count of things tried did.

What it would take to clear the higher bar

At a 60% win rate with equal-sized wins and losses, a single test clears break-even after about 97 trades. At the best-of-forty bar it takes about 261. The trades-needed calculator and the win-rate interval calculator both use the single-test 95% bar, so treat their answers as the floor for a result picked from a sweep, not the target.

The objection, and what is left of it

A 19-period lookback and a 20-period lookback take nearly the same trades, so forty neighbouring variants are fewer than forty independent tests, and Bonferroni charges too much. That is true. It changes the size of the penalty, not its direction: the honest z sits somewhere between 1.96 and 3.23, and 1.96 is the one value it is not. The longer version is in the note Best of forty is not a result.

How to run the check on any report

1. Ask for the number of variants. Parameter combinations, filters, window edges and abandoned ideas. If the report cannot say, assume it is more than one.

2. Move the bar. Divide 5% by that number and use the matching z. Forty variants is about 3.23; a hundred is about 3.48.

3. Ask for the whole table, sorted. A setting that works usually sits on a plateau: its neighbours do nearly as well, and the median variant is not a disaster. A lucky setting is a spike, one row standing alone above a flat field.

4. Test the winner on data the sweep never saw. A later window, or a forward record with the setting frozen. That is check 10.

This is check 9 of 10

The one-page Backtest Verification Checklist, free. Confirm the email and the PDF opens straight away; one worked check each Tuesday after that.

How these numbers were worked out

Every figure is a made-up sweep, not a result of any system. Intervals are Wilson score intervals on the stated wins and trades: at z = 1.96, 120 of 200 gives 53.1% to 66.5%; at the Bonferroni z, 120 of 200 gives 48.6% to 70.4%. The Bonferroni z is the two-sided normal quantile for p = 0.05 ÷ 40 = 0.00125, which is 3.227; for 100 variants, p = 0.0005 gives 3.48. Break-even with equal-sized wins and losses is 1 ÷ (1 + 1) = 50%. The chance that at least one of forty no-edge tests passes at 5% is 1 − 0.9540 = 0.871. The expected best of forty is 50% plus 2.161 standard errors, where 2.161 is the expected largest of 40 standard normals (numerical integration) and one standard error at 200 trades is √(0.25 ÷ 200) = 3.54 points: 50% + 7.64 = 57.6%, or about 115 wins. The chance one no-edge variant wins 120 or more of 200 is the exact binomial tail, 0.00284; for forty independent variants, 1 − (1 − 0.00284)40 = 0.108. “Trades needed” is the smallest count at which the lower end of the interval sits above break-even, assuming the 60% holds and trades are independent, the same search the calculator runs: 97 at z = 1.96, 261 at z = 3.227. Nothing here is investment advice.

Historical simulations are not live results. Nothing on this site is investment advice.