A strategy report shows one configuration. The question it almost never answers is how many were tried before that one was chosen. This note is the arithmetic of that question, and it is harsher than most people expect.
Take a system with no edge at all, so each variant’s t-statistic is a draw from a standard normal distribution. Test one variant against the usual two-sided 5% bar and it clears 5% of the time, by construction. Test twenty and the chance that at least one clears is 64%. Test forty and it is 87%. Test a hundred and it is 99%.
chance at least one clears = 1 − 0.95^N
Nothing in that calculation involves a good idea. It is what happens when you keep drawing.
The same question from the other side: given N worthless variants, how large is the largest t you should expect? For twenty variants the expected maximum is about 1.87. For forty it is about 2.16. For a hundred, about 2.51.
So a reported t of 2.0 chosen from forty attempts is below what noise produces on average. That is not weak evidence. It is the absence of evidence wearing a result’s clothes.
Searching is not forbidden. It just reprices the threshold. Bonferroni divides the family error rate by the number of tests: at forty variants and a 5% family rate, the winner needs p < 0.05 / 40 = 0.00125, which in large samples is a t of about 3.23. The Šidák version, 1 − (1 − 0.05)^(1/40) = 0.00128, puts it at about 3.22, near enough the same place.
That is the real price of a parameter sweep. The winner has to clear roughly three and a quarter, not two.
N is every configuration tried, not every configuration reported. Three kinds go missing.
A report that cannot tell you N cannot tell you what its p-value means.
Variants of one strategy are not independent. A 19-period lookback and a 20-period lookback take nearly the same trades, so forty correlated variants contain far fewer than forty independent tests, and a Bonferroni correction at N = 40 over-penalises. That is true, and it is why the correction is called conservative rather than correct.
What the objection changes is the size of the penalty, not its direction. The effective number of tests is smaller than the raw count and larger than one, so the honest bar sits somewhere between 1.96 and 3.23, and 1.96 is the one value it certainly is not. If you want the exact figure rather than a bracket, bootstrap the whole set of variants together, which prices the correlation instead of assuming it away.
The useful artefact from a sweep is the whole set of results, sorted, not the top of it. A sweep worth something usually looks like a plateau: the neighbours of the chosen setting perform nearly as well, and the median variant is not a disaster. A sweep worth nothing looks like a spike, one configuration standing alone above a flat and unprofitable field.
That chart takes a minute to produce, and it answers the question the headline figure was selected to avoid.
Count the attempts, move the bar, then look at the figure. Done the other way round, figure first and attempt count recalled afterwards, the figure always survives, because by then you are explaining it rather than testing it.
Every figure above is arithmetic on a system with no edge, computed for this note: the probabilities as 1 − 0.95^N, the expected maxima by numerical integration of the largest of N standard normals, and the bars as two-sided normal quantiles. None of it is a result of mine.
These are the same checks I run before I believe any backtest, including my own.
Historical simulations are not live results. Nothing on this site is investment advice.