How we test the trading rules

What we check when we build a trading rule, and what we check every day while it runs.

1. Checking that rules set on the past still work afterwards

We set the rules on past data, then measure the results on the period that follows. Repeating this across the whole history is walk-forward testing.

Trading costs such as commissions and slippage are included.

Walk-forward testing does not mean every design choice is independent of the data in the test periods. The session each asset trades, when each asset joined the portfolio, the entry filters and exit rules, the risk multipliers and some learning settings were decided after looking at the full period. See the walk-forward record →

2. Checking it holds up when conditions change a little

• We check the result on different charts.
  The same market's price data differs by data vendor, by micro or mini contract, and by how contract months are joined. We test again on the other data and see how much the results change.
• We check the result when prices are a little different.
  We also test on synthetic data with prices deliberately nudged. We are wary of a strategy whose results swing on a few ticks.
• We check it is not overly sensitive to its settings.
  We change the main settings a little at a time and compare the results. A strategy that does especially well at one particular value only is not adopted. See parameter sensitivity →
• We test under unfavourable conditions, not only favourable ones.
  Instead of the settings with the best result, we choose the ones that hold up relatively well when conditions get worse. See the robustness tests →
• We check the results do not change much when the randomness of training or the starting point changes. See training seed robustness →

3. Checking it is not just luck

• We compare against random trades.
  We build a control group whose trade direction is chosen at random and compare it with the real strategy, to see whether the real result is at a level chance alone could produce. See the random direction control →
• We check the periods separately.
  We split the whole record into an earlier and a later period and look at each, to see whether the results are concentrated in one period. See results by period →
• We check the profit is not concentrated in a few trades.
  We look separately at whether a particular year or a handful of large winning trades made most of the overall result. See the best-year share →

Pass criteria are set, and the results are also reviewed separately from the point of view of refuting them. When a change is adopted although it fell short of a criterion, that fact goes into the change log.

4. Testing under real trading conditions

Backtest numbers alone are not enough. We also account for the costs of real trading and for account rules.

• Commissions and slippage are included, and we also test separately with larger slippage. See results by execution cost →
• Results are calculated in whole contracts that can actually be traded.
• For prop accounts, pass or fail is judged by applying each firm's official rules to day-by-day results, including loss limits and consistency rules. See the prop simulator →

The intraday loss-limit check is an approximation and can differ from the firm's own ruling.

5. Watching first when a rule changes

Where possible, a new rule is not applied straight away. It runs next to the existing rule, recording its signals, and we look at the results before applying it. We call this a shadow run.

A change with a short shadow run, or none, is not hidden: it is recorded in the change log.

We also record what changed and when, and how the results changed afterwards. See the change log →

6. Checking every day while it runs

Testing does not end at launch. Every day we check that the signals actually sent are produced by the rules that were tested.

• Rule match: An automatic check compares whether the actual signals went through the same decision process as the rules used in testing.
• No future information: We check that no information that was unknown when the signal was made went into the decision.
• Record integrity: The record of sent signals is sealed every day with a certificate from an independent timestamp authority. If a record were changed afterwards, the change would show. See the sealed records →

We publish not only the test results but also the signals actually sent and the trading results. Winning and losing trades can both be checked. See the architect's trading account record →

7. Failed experiments stay on record

Experiments that did not pass the test criteria are not deleted.

Keeping the failed attempts means that when an idea is tested again the earlier result can be checked, and what did not work can be tracked.

Past performance does not guarantee future results.