/ Check 5 · Circular bootstraps

Does a Monte Carlo test prove my backtest?

Usually not. A Monte Carlo run on the backtest's own trades re-describes those trades; it cannot tell you whether the edge behind them is real. Here is why, with one worked example and no code.

The short answer

Most backtest reports that say “passed 1,000 Monte Carlo runs” took the backtest's own trade list and reshuffled or resampled it. That test is circular: its only input is the result it is meant to check. It can describe how bumpy the same trades could have been in a different order. It cannot say whether those trades would happen again, because nothing new went into it.

A worked example: 50 trades, 1,000 reshuffles

Take a backtest with 50 trades, an average of +0.2R a trade (R is the amount risked on each trade) and a standard deviation of 1.2R. The total is 50 × 0.2 = +10R.

  • Reshuffle the order 1,000 times. Every one of the 1,000 runs ends at exactly +10R, because a sum does not depend on the order of its terms. “100% of simulations profitable” is guaranteed before the first run. What changes between runs is the path: the drawdowns and losing streaks. That is useful for sizing, and it says nothing about whether the edge exists.
  • Resample with replacement instead. Now the totals do vary, but they are centred on the same +0.2R average the backtest already reported. The spread of that average is the standard error, 1.2 ÷ √50 ≈ 0.17R. On the normal approximation, about 12% of resampled averages land below zero, and the 95% range for the average runs from about −0.13R to +0.53R.

So the honest reading of this bootstrap is “at 50 trades, we cannot tell this edge from zero”, the same thing a plain confidence interval says without any simulation. And even that reading is the best case: it assumes the 50 trades are a fair sample of the future. If they were chosen, the bootstrap inherits the choice.

Why “circular”

Suppose the 50 trades came from the best of many parameter settings. The settings were picked because these trades looked good, so resampling them measures the luck of the winner, not the edge of the method. The test uses the same data twice: once to choose the strategy and once to grade it. A bootstrap is only as independent as its input. Fed the in-sample trades of an optimised strategy, it will agree with the optimiser every time.

Three questions to ask of any Monte Carlo claim

1. What was resampled? If the answer is “the backtest's trades”, the test can only speak to path risk (drawdown, streaks), never to whether the edge is real. A reshuffle always lands on the same total, so its “share of profitable runs” is 100% or 0% by construction and carries no information.

2. Were those trades selected? Ask how many settings, markets or rules were tried before this one. If more than one, the bootstrap needs the trades of every candidate, or out-of-sample trades the selection never saw.

3. Does the average clear zero by enough? Divide the average trade by its standard error. In the example, 0.2 ÷ 0.17 is about 1.2, well short of the 2 that a 95% bar needs. At the same average and spread it would take about 144 trades to reach 2. The trades-needed calculator and the win-rate interval do the same arithmetic for win rates.

What replaces it

Data the strategy has never seen. A walk-forward test, a later period held back before any tuning, or a live forward record (check 10) all add information the backtest did not have. A bootstrap on those trades is a real test. A bootstrap on the trades that made the strategy look good is a description of them.

This is check 5 of 10

The one-page Backtest Verification Checklist, free. Confirm the email and the PDF opens straight away; one worked check each Tuesday after that.

How these numbers were worked out

Every figure is arithmetic on one made-up trade list, not a result of any system: 50 trades, mean +0.2R, standard deviation 1.2R. Total = 50 × 0.2 = 10R. Standard error = 1.2 ÷ √50 = 0.170R. The share of resampled means below zero is the normal tail below −0.2 ÷ 0.170 = −1.18, which is 11.9%. The 95% range is 0.2 ± 1.96 × 0.170, or −0.13R to +0.53R. Trades for a ratio of 2 = (2 × 1.2 ÷ 0.2)² = 144. A real bootstrap of a skewed trade list will differ a little from the normal approximation. Nothing here is investment advice.

Historical simulations are not live results. Nothing on this site is investment advice.