The backtest baseline is what everything else is measured against: the strategy with no filter, no extra rule, no intervention. When you say "this filter adds 5 percentage points", it adds them relative to the baseline. If the anchor has drifted, every statement built on it has drifted too — and all in the same direction.
Ours had drifted. Of 4,232 runs counted as "no filter" in our Edge Library, 2,370 were not filter-free runs at all — they had a minimum-profit guard switched on. 61.6 percent. We recomputed the aggregation, and while doing so changed a second thing that was not a bug. Together they flip 20 of 352 verdicts. This piece shows what happened, how we separated the two causes, and what rule came out of it.
The defect: a column that does not exist
Our backtest runs store a marker for each filter: 200-week average on or off, volatility filter low/high/expanding, altcoin season on or off, bull-market gauge in three stages. The aggregation reads those markers and sorts every run into its group.
For the minimum-profit guard there is no such marker. It gets set, it takes effect, it changes the result — but it leaves no trace in the database that the aggregation could read. So a run with the guard active looked, to the aggregation, exactly like a run with no filter at all. It landed in the baseline.
The size of that is measurable:
| Group | Runs | Median CAGR |
|---|---|---|
| true baseline | 1,862 | 0.00 % |
| minimum-profit guard | 2,370 | −3.75 % |
| published anchor | 4,232 | −1.85 % |
The anchor sat 1.85 percentage points too low. And since every filter advantage is reported as a difference to that anchor, every reported advantage was overstated by exactly that much. Filters looked systematically better than they were.
Per strategy it gets sharper, because the mix varies:
| Strategy | True baseline | Published anchor | Shift |
|---|---|---|---|
| wma_trend | 14.0 % | −2.5 % | −16.5 pp |
| regime_200d | 21.6 % | 12.3 % | −9.3 pp |
| btc_signal_rsi_sma | 19.8 % | 12.0 % | −7.8 pp |
| ema_trend_bias | 0.5 % | −5.1 % | −5.6 pp |
Two smaller siblings of the same problem came with it. The "bull market (early)" variant matched none of the eight group rules and dropped out of the aggregation entirely — 2,003 runs that appeared nowhere. And the weekly computation read runs from the database page by page without requesting a fixed sort order. Without a sort, the database guarantees no stable row order between pages: rows can arrive twice, others not at all. So the published numbers were computed weekly from a subset nobody knew.
The choice: which candle intervals sit in the pool
The second change was not a defect, and that is the more interesting part.
Our Edge Library groups all runs per market, strategy and filter, then takes the median. "All runs" means: across all candle intervals — daily, weekly and monthly together in one pool. Fill the pool differently and you get a different median. Perfectly legitimate, as long as you know what is in it.
We did not know precisely. Measured, the pool held only 1-day, 1-week and 1-month candles. The 2- and 3-day intervals our interface has always offered had never been run by the automated aggregation — exactly zero cells. Which meant the two long-horizon intervals, the ones that produce the fewest signals on short price histories, carried disproportionate weight.
That is not a trifle: it meant an undeclared decision — which intervals count — shaped every published figure without being written down anywhere. We recomputed 15,152 backtests and unified the base to 1d/2d/3d/1w/1M.
The old numbers were correct under their own choice. We now choose differently, and we say so.
Why we measured both separately
Two changes hit the same numbers. A plain before-and-after would have shown only the sum — leaving open whether the movement came from a defect or from a decision. That is a difference a reader should know.
So two runs instead of one: the old aggregation code on the new data isolates the interval effect. The new code on the same data isolates the classification effect. Both compute on a single loaded set of rows, so the difference is guaranteed to be the code alone.
| Cause | Verdicts flipped | DSR outcome flipped | Deltas moved |
|---|---|---|---|
| Interval base (choice) | 18 | 30 | 99 |
| Baseline correction (defect) | 10 | 0 | 164 |
| total | 20 | 30 | 164 |
The rows do not add up: if a verdict flips in the first step and back in the second, it counts twice individually and not at all in the total.
The ratio is the notable part. The larger visible movement comes from our decision, not from the defect. Writing only "we fixed a bug" would have hidden the bigger half.
What changes for readers
Four filters reverse sign. We now say the opposite of what we said before:
| Strategy / filter | before | after |
|---|---|---|
| ema_cross + ATR expansion | helps (+1.8 pp) | hurts (−2.9 pp) |
| ema_trend_bias + bull market (strict) | helps (+5.3 pp) | hurts (−2.3 pp) |
| ema_trend_bias + bull market (confirmed) | helps (+1.5 pp) | hurts (−2.2 pp) |
| fg_cadence + 200-week average | helps (+1.4 pp) | hurts (−1.6 pp) |
On top of that, 88 new rows: minimum-profit guard and bull market (early) are now entries of their own instead of vanishing into the baseline or missing entirely.
And the number of tested configurations rises from 7 to 9. That sounds like a footnote but it is the hardest consequence: the more variants you test, the more likely you are to find a good-looking one by chance alone. The correction for that — the Deflated Sharpe Ratio — gets stricter with each additional variant tested. 30 cells now report a result where "not computable" stood before, and they do not pass it. That is not the strategies getting worse; it is a statement that did not exist before.
Limits of this analysis
Three things we cannot show:
The interval effect and the pagination defect cannot be cleanly separated. The frozen before-state came out of the faulty weekly run, and the old data situation no longer exists. Only the classification effect is cleanly isolated. So the 18 flips in the first step contain both.
For runs without a filter marker, the minimum-profit guard stays invisible. We now read the marker where it exists — for runs from the web interface it does not, so the derivation from the filter columns still applies there, and the guard does not show up in it. That is the remainder of the old defect. Closing it fully requires a new column.
The 2- and 3-day cells cover a slightly longer period than their daily siblings. They were computed on a single day, while the older cells accumulated across 67 rotation days. For a median over dozens of pairs this barely registers, but it is not zero.
The rule that came out of it
The real finding sits underneath the numbers. Raw data has objectivity: a close is a close. Derived numbers do not — they have a method, and the method has decisions. A "median CAGR of strategy X" depends on the pair selection, the intervals, the period, gross or net, the weighting, and the version of the computing logic. None of that is given by nature.
And a decision written down nowhere will eventually be re-made silently. That is precisely what creates the impression of a platform correcting itself in circles — when nobody was actually wrong.
We turned that into a binding rule. Before any change to a published number, we write down what the intervention is:
| Defect | Method decision | |
|---|---|---|
| Recognised by | There is verifiable evidence: a code difference, a source date, a reproduction. The old number was demonstrably wrong. | The old number was not wrong. A parameter is chosen differently. |
| Handling | Gets fixed, evidence documented. | Needs justification, versioning and a changelog entry. The old number stays traceable. |
| Reversible? | Only with new counter-evidence. | Yes — it is a choice. Which is exactly why it must be marked as one. |
When both hit the same number, the contribution is decomposed rather than summed. That is what happened above.
The gap this rule closes is precise: a method decision must not pass as a bug fix. Otherwise it is repeatable in both directions, as often as you like, with nobody accountable.
For the backstory: we already recomputed one of our own histories in July, back then because of look-ahead bias in macro data. And why we set backtests up the way we do is in why honest backtesting looks different.
What Backtesting Arena contributes here
The Edge Library compares strategies with and without filters across a fixed universe, corrects for multiple testing, and reports sample sizes. Which makes it exactly the kind of analysis that can be quietly wrong without anyone noticing — after all, there is a number on the screen.
What we take from this is less the correction than the machinery behind it: checks that fire when a group membership shifts; a test that holds the pipeline's filter variants against the aggregation's groups; a fixed sort order when reading page by page. The defect above stayed invisible because no check existed that could have made it visible. Now one does.
Both aggregation states — before and after — are stored as files in the project. Anyone who wants to look up an old number can.
Frequently asked questions
So were the old numbers wrong? Partly. The baseline anchor was demonstrably wrong — that is a defect. The interval composition was not wrong but undeclared; we changed it. Of the 20 flipped verdicts, 18 trace to the interval choice and 10 to the anchor correction, some counting in both.
Why did nobody notice earlier? Because a wrong baseline does not announce itself: it produces no error, no blank page, no absurd figure. It produces a plausible number that is off by a constant amount. It surfaced only when we held the pipeline's filter variants against the aggregation's groups — 11 against 8, and the difference was the answer.
Will this change again? The defect will not, absent new counter-evidence. The method decisions may — whether the two altcoin-season modes are reported separately or together, for instance, is open. Such changes will carry a justification and a changelog entry rather than appearing as a correction.
Does "hurts" mean I should not use these filters? No. A negative median CAGR across a pair universe means the filter cost return on average. It may still have reduced drawdown or cut the number of false signals — both reasons to use it. The Edge Library provides evidence, not recommendations.
Why does the number of tested configurations rise, and what does that change? Because two variants were added that were previously invisible. The more variants you test, the more often you find one that looks good by chance. The Deflated Sharpe Ratio prices that in, and gets stricter with every additional variant. That 30 cells now fail it follows from more accurate bookkeeping, not from worse strategies.
How does this affect running backtests? Not at all. The individual backtest results are unchanged — only how they are grouped and compared with each other has changed.