🎯 In 30 seconds: An idea is only interesting if it does better than what happens anyway, which is the base rate. With small samples the error is so wide that most results mean very little. Settings that only work at one exact value are usually luck. Every extra idea you try raises the odds that one looks good by chance. Costs measured in R can wipe out a thin edge, especially with tight stops.
Question one: compared with what?
Suppose a hypothetical stock rose over the five days after a hammer candle 58% of the time. That sounds useful until you ask how often it rose over any five days.
That second number is the base rate. If the stock rose over 54% of all five-day windows, the hammer added only 4 points. In a long uptrend, almost any signal looks good because the stock went up anyway.
The same thinking applies to returns. A rule that made 30% while the stock itself gained 45% over the same period did worse than simply holding it, even though it made money.
The site’s Pattern record tab is built around this comparison. It sets what followed each candle pattern on a stock beside the base rate for that stock.
Question two: how sure can you be?
A win rate from a small sample is fuzzy. A rough 95% margin of error for a win rate near 50% is about 1.96 × √(0.25 ÷ n), where n is the number of trades.
| Trades (n) | Margin of error, win rate near 50% | A measured 58% could really be |
|---|---|---|
| 20 | ± 22 points | 36% to 80% |
| 50 | ± 14 points | 44% to 72% |
| 100 | ± 10 points | 48% to 68% |
| 400 | ± 5 points | 53% to 63% |
| 1,000 | ± 3 points | 55% to 61% |
With 50 occurrences, a 58% hammer and a 54% base rate cannot be told apart. You would need several hundred independent cases before a 4-point gap starts to stand out.
Counting independent cases
Signals often come in clumps. A breakout rule might fire three days in a row during one move. Those are one event counted three times.
The same goes for testing one idea on ten tech stocks during one rally. They moved together, so they behave more like one test than ten. Spread tests across different years, sectors and market conditions, including a falling market.
Overfitting: when settings fit noise
Every rule has settings: the look-back length, the stop multiple, the exit. Trying many values and keeping the best fits the rule to the quirks of that one history.
A practical test is to look at the neighbors. If a 50-bar setting works, 40 and 60 should work almost as well. A result that only exists at one value is a spike, and spikes rarely repeat.
Counting your tries
Statistical tests usually allow a 5% chance of a false positive. Test 20 unrelated ideas that truly do nothing, and on average one will pass by luck. The chance that at least one passes is about 64%.
So the number that matters is not only how good the winner looks. It is how many ideas, settings and stock lists you tried to find it. Sullivan, Timmermann and White (1999) showed how a best rule picked from thousands can look strong in one period and lose its significance once that search is accounted for. Bailey and coauthors (2014) made the same point about how easily backtests can be overfit.
Costs in R
Costs are often written as a percent of the trade. It helps to convert them into R, because R is how the edge is measured.
Say the spread plus slippage costs 0.2% of the price for a round trip. If the stop is 2% away, that cost is 0.2 ÷ 2 = 0.1R on every trade, win or lose.
| Stop distance | Round-trip cost 0.2% | A 0.15R gross edge becomes |
|---|---|---|
| 1% of price | 0.20R per trade | −0.05R (losing) |
| 2% of price | 0.10R per trade | +0.05R |
| 4% of price | 0.05R per trade | +0.10R |
| 8% of price | 0.025R per trade | +0.125R |
Tight stops feel safe but make costs a large share of each trade. Short-term ideas on intraday bars feel this the most. Taxes come on top and depend on your situation.
A testing routine that fits on one card
- Write the rule, the stock list and the dates before looking at any result.
- Enter at the next bar’s open after the signal, never at the signal bar’s close.
- Hold back the most recent third of the data. Do not look at it yet.
- On the older two thirds, try a few settings and write each one down.
- Compare the chosen version with the base rate and with holding the stock.
- Charge costs in R. Check that neighbors of your setting also work.
- Run the chosen version once on the held-back data.
- Paper trade it forward on new bars, unchanged.
For the rules you might test, see Five classic setups. The chapter Does technical analysis work? covers the research history.
- A signal is followed by a rise 60% of the time. What do you need before calling that good? Answer: The base rate: how often the stock rose over the same window on any day.
- Roughly how wide is the margin of error on a win rate from 20 trades? Answer: About 22 points either way.
- A rule works at a 50-bar setting but fails at 45 and 55. What does that suggest? Answer: The result is probably a lucky spike, not a real effect.
- Costs are 0.2% per round trip and the stop is 1% away. What do costs take per trade in R? Answer: 0.2R.
Sources
- David Aronson, Evidence-Based Technical Analysis (2007), on data-mining bias and testing rules.
- Sullivan, Timmermann and White (1999), “Data-Snooping, Technical Trading Rule Performance, and the Bootstrap,” Journal of Finance.
- Bailey, Borwein, López de Prado and Zhu (2014), “Pseudo-Mathematics and Financial Charlatanism,” Notices of the American Mathematical Society.
- Park and Irwin (2007), “What Do We Know About the Profitability of Technical Analysis?” Journal of Economic Surveys.
- Perry J. Kaufman, Trading Systems and Methods, on system testing and costs.
- Moneydesks Technical Analysis tool: the Pattern record tab and base rate.