Start with a number most reports leave out: how many settings were tried before this one was picked. The best of forty is held to a higher bar than one test run once, and the usual 95% interval does not know the other thirty-nine existed.
Every optimizer has a best row, whether or not anything in the sweep works. A result picked from a sweep has to clear a bar that rises with the number of settings tried. Each result also has to look like its neighbours. If you do not know how many settings were tried, you do not know what the reported confidence means.
A made-up strategy is swept over 8 lookbacks and 5 stop distances, forty variants, each producing 200 trades with wins and losses about the same size. Break-even at that payoff is 50%. The best variant wins 120 of 200, a 60% win rate.
Taken on its own, that looks like a result. The 95% interval on 120 of 200 is 53.1% to 66.5%, entirely above break-even. Now give it the history it actually has.
So the same 120 of 200 passes as a single test and fails as the best of forty. Nothing about the trades changed. Only the count of things tried did.
At a 60% win rate with equal-sized wins and losses, a single test clears break-even after about 97 trades. At the best-of-forty bar it takes about 261. The trades-needed calculator and the win-rate interval calculator both use the single-test 95% bar, so treat their answers as the floor for a result picked from a sweep, not the target.
A 19-period lookback and a 20-period lookback take nearly the same trades, so forty neighbouring variants are fewer than forty independent tests, and Bonferroni charges too much. That is true. It changes the size of the penalty, not its direction: the honest z sits somewhere between 1.96 and 3.23, and 1.96 is the one value it is not. The longer version is in the note Best of forty is not a result.
1. Ask for the number of variants. Parameter combinations, filters, window edges and abandoned ideas. If the report cannot say, assume it is more than one.
2. Move the bar. Divide 5% by that number and use the matching z. Forty variants is about 3.23; a hundred is about 3.48.
3. Ask for the whole table, sorted. A setting that works usually sits on a plateau: its neighbours do nearly as well, and the median variant is not a disaster. A lucky setting is a spike, one row standing alone above a flat field.
4. Test the winner on data the sweep never saw. A later window, or a forward record with the setting frozen. That is check 10.
The one-page Backtest Verification Checklist, free. Confirm the email and the PDF opens straight away; one worked check each Tuesday after that.
Every figure is a made-up sweep, not a result of any system. Intervals are Wilson score intervals on the stated wins and trades: at z = 1.96, 120 of 200 gives 53.1% to 66.5%; at the Bonferroni z, 120 of 200 gives 48.6% to 70.4%. The Bonferroni z is the two-sided normal quantile for p = 0.05 ÷ 40 = 0.00125, which is 3.227; for 100 variants, p = 0.0005 gives 3.48. Break-even with equal-sized wins and losses is 1 ÷ (1 + 1) = 50%. The chance that at least one of forty no-edge tests passes at 5% is 1 − 0.9540 = 0.871. The expected best of forty is 50% plus 2.161 standard errors, where 2.161 is the expected largest of 40 standard normals (numerical integration) and one standard error at 200 trades is √(0.25 ÷ 200) = 3.54 points: 50% + 7.64 = 57.6%, or about 115 wins. The chance one no-edge variant wins 120 or more of 200 is the exact binomial tail, 0.00284; for forty independent variants, 1 − (1 − 0.00284)40 = 0.108. “Trades needed” is the smallest count at which the lower end of the interval sits above break-even, assuming the 60% holds and trades are independent, the same search the calculator runs: 97 at z = 1.96, 261 at z = 3.227. Nothing here is investment advice.
Historical simulations are not live results. Nothing on this site is investment advice.