Why Your Backtest of a News Strategy Was Wrong

There is a specific disappointment that most people who trade news eventually meet. The idea tested well. The curve went up and to the right, the sample was not tiny, and the rules were simple enough that there was nothing obvious to overfit. Then it went live and the results were not merely worse — they were a different shape entirely. Trades that the test said were small winners came back as small losers, and the edge quietly inverted.

The usual diagnosis is overfitting, and overfitting is real. But it is not the explanation here, and reaching for it first means you fix the wrong thing. There is a more specific and more mechanical reason a news strategy in particular fails the transition from test to live, and it is the thing the last two pieces on this blog have been circling: a backtest replays the price on the chart, and the price on the chart is not a price you could have traded.

The one assumption underneath all of it

Historical bar data is a compression. Each bar keeps an open, a high, a low and a close, and it throws away everything else that happened inside it — the order in which prices arrived, how much size stood behind each one, what the other side of the market was quoting, and whether anyone would have dealt with you at all. A backtest works by pretending that what survived the compression is enough to reconstruct a trade.

In quiet conditions that pretence is close enough to true that it does not matter much. Around a release it stops being true in several ways at once, and — this is the part that actually causes the damage — every one of those ways is biased in the same direction.

Five places the fill and the chart come apart

Worth naming them separately, because they have different fixes and people tend to be aware of one and blind to the other four.

• The chart is one side of the market. Most platforms chart the bid, some the mid. If you are buying, you pay the ask, which was never on the chart your test read. The test bought at a price that was not for sale.
• The spread is missing from the data entirely. Bar history does not record it, so the test had to assume one — usually implicitly, usually a quiet-hours figure, and usually applied identically to every trade including the ones taken in the worst seconds of the day.
• Slippage into a thin book. A market order does not get the price it saw; it gets whatever is standing there. In the window a news strategy trades, the least is standing there.
• Order sequencing inside a bar is unknowable. If one bar touched both your stop and your target, the data cannot say which came first. The backtest resolves that ambiguity with a rule, and the rule is optimistic more often than not.
• The fills you would never have received. Rejections, requotes, a platform that hesitated, and the plain fact that a human needs a moment to act. The test fills every order instantly and unconditionally.

Each of these is well known in isolation. What makes news strategies a special case is not that they suffer from a sixth, exotic problem. It is that they take all five at maximum strength, on purpose, on every trade.

Why this is a bias and not just noise

This is the distinction that changes what you should do about it. If execution error were random — sometimes better than the chart, sometimes worse — it would be noise. Noise widens your distribution of results but leaves the average roughly where the test put it, and a large enough sample would average it away. That is the intuition most people are running on, and it is why the response to a disappointing live result is so often 'give it more trades'.

But none of the five is symmetric. You always pay the spread rather than sometimes collecting it. Slippage on a market order into a thin book is not equally likely to help you. An ambiguous bar resolved in your favour by the test is resolved by the market on its own terms. The order that would have been rejected is missing only from the test. Add five one-directional errors together and you do not get noise; you get a systematic overstatement of the result, and more trades make it more visible rather than less.

This also explains the specific shape of the disappointment. The strategy's big winners usually survive, because a large move dwarfs the execution cost. What disappears is the middle of the distribution — the modest winners that were only modestly better than the cost of trading. When those flip, the average flips with them, and it does so without any single trade looking obviously wrong.

How to find out what it costs you, specifically

The instinct at this point is to fix the backtest — better data, a slippage parameter, tick history. That helps, and it is worth doing, but it cannot settle the question, because you would still be choosing the numbers you feed it. The reliable move is the other direction: stop asking history what the strategy would have made and start recording what it can actually get.

This is a forward log, not a backtest, and the whole point is that it is collected in real time on your own account, platform and instrument. For each signal the strategy generates — whether or not you take it — record:

• The chart price at the moment the signal triggered. This is the number the backtest would have used.
• The actual bid and the actual ask at that same moment, so you can see which side you would really have dealt on and how far apart they were.
• The time you could genuinely have acted, not the timestamp of the trigger. If the rule needs a bar to close and then a decision, that delay is part of the strategy.
• The price actually available at that later moment — the honest entry.
• If you took the trade: the fill you received, and the difference from the price you clicked.

Two numbers fall out of this and they are the ones worth having. The first is the gap between the backtest's assumed entry and your honest entry, averaged over your own signals. The second is how that gap varies — because a strategy whose cost is stable is a different proposition from one whose cost occasionally triples in exactly the conditions it likes to trade.

The ambiguous-bar audit

There is one more check, and it takes an afternoon rather than weeks. Go back through the trades in your backtest and count how many had a bar that touched both the stop and the target. For each of those the test made a choice the data could not support. The fraction of your results that rests on that choice is the size of the assumption you have been carrying, and it is usually larger than people expect for strategies with tight stops around volatile moments.

If that fraction is small, you can stop worrying about it. If it is not, re-run the test with the ambiguity resolved against you every time. The gap between the two runs is not a prediction — the truth is somewhere between them — but it brackets the answer honestly, which is more than the original number did.

The three honest responses

Suppose you do this and the edge is thinner than advertised, or gone. There are only really three moves, and it is worth knowing them in advance so the discovery does not turn into improvisation.

The first is to re-test with your measured costs instead of assumed ones and see what survives. Some strategies still work with a realistic cost applied; they are just less impressive than they looked. That is a perfectly good outcome and by far the most common one.

The second is to move the trade out of the expensive window. This is where the previous two pieces connect: if the entry does not have to happen in the seconds around the print, waiting for conditions to normalise turns a maximum-cost trade into an ordinary-cost one. The strategy changes, so it needs testing again — but it changes in the direction of being executable.

The third is to accept that the edge exists but not at your size, instrument or account type, and to leave it alone. This is the least popular answer and it is a real one. A cost you cannot avoid is a constraint, not a problem to be solved by trying harder.

What this does not mean

It would be easy to read all of this as 'backtesting is pointless', and that would be the wrong lesson. A backtest is very good at one job: ruling things out. If an idea does not work on clean historical data with generous assumptions, it will certainly not work with realistic ones, and you have saved yourself the live experiment. That is genuinely valuable and it is cheap.

What a backtest cannot do is certify. Passing the test moves an idea from 'disproven' to 'not yet disproven', and the remaining work — the part that decides whether there is any money in it — happens in front of live prices with a notebook open. Treating the test as the end of the process rather than the first filter is the actual mistake, and it is a much more comfortable mistake to make than sitting through a few weeks of forward logging.

FAQ

My data includes bid and ask ticks. Am I safe?

Better positioned, and it removes the first two problems on the list outright, which is most of the arithmetic. It does not remove the rest. Tick data records the prices that were quoted, not how much size stood behind them, so slippage on anything beyond a small order is still an assumption. It also cannot represent your own latency — the seconds between the condition being met and you or your platform acting — or the orders that would have been rejected. Bid-ask tick data turns a large unknown into a smaller one; it does not close the gap.

Can I just subtract a fixed cost per trade?

It is much better than assuming zero, and if you do nothing else, do this. The limitation is in the word fixed. The cost of trading is not a constant that the market charges you; it is largest precisely when conditions are worst, which for a news strategy is precisely when it trades. A flat deduction taken from a quiet-hours figure will flatter exactly the strategies that most need the correction. If you use a fixed number, take it from your own forward log around releases rather than from a broker's typical-spread page.

How many forward trades before I trust the number?

Deliberately not answered with a figure, for the same reason the spread piece refused to define 'materially wider' — any number would be wrong for most readers, and quoting one invents a precision nobody has. The useful stopping condition is behavioural rather than numerical: keep logging until new observations stop moving your cost estimate. When another week of signals leaves your average roughly where it was, you have enough for that instrument and that time of day. If it is still jumping around, you have learned something more important than the average, which is that your cost is unstable.

Does any of this apply to a strategy that avoids news?

Yes, but much less severely, and that is the whole point of the comparison. Every strategy pays the spread and takes some slippage; a strategy trading ordinary conditions pays a small, stable version of it, so the backtest's error is small and roughly constant. The one caveat is that 'avoids news' deserves checking rather than assuming — a strategy with no view on the calendar will sit in trades through releases without meaning to. Comparing your trade timestamps against a calendar for a month usually answers it quickly.

Is paper trading enough, or do I need real money?

Paper trading answers some of this and not the rest, and it is worth being precise about which. It genuinely captures the chart-versus-quote gap, the spread at the moment of the signal, and your own reaction time — three of the five, and the three that are easiest to get wrong on paper. It does not capture slippage or rejection honestly, because a simulator will usually fill you at the quote regardless of size. So a paper forward-log gives you a good measurement of your entry gap and an optimistic one of your fill. That is still far more than the backtest gave you, and knowing which half is optimistic is what stops it from becoming the next false confidence.