Finding Out What Works, And What Doesn’t Work

Many traders who try system trading have previously had difficulty at discretionary or manual trading. Most of these folks eventually recognize the benefit of trading a system with well defined rules - a system that has performed well in the past. It is nice to know a trading approach has historically worked, but as with all things related to trading, past performance is no guarantee of future results.

Unfortunately, many people who try systematic/algorithmic/mechanical/rule-based trading for the first time bring along a lot of the baggage that they have acquired from their previous method. Depending on the pre-conceived notions they bring into mechanical trading, these new systematic traders may run into a lot of frustration and trouble.

Many times, for example, traders will always test with a few core concepts, such as always closing by the end of the day. This is what they were used to as a discretionary or manual trader, and therefore they never even think to test ideas out of their old comfort zone. Perhaps removing these "comfort" rules would dramatically improve performance.

In this article, I will examine three common items that new systematic traders test, and see how these items actually work when they are subjected to rigorous testing.

Ground Rules

In all the testing that follows, I will test in 7 different markets, across a range of commodities:

  • Wheat (W)
  • 10 Year Treasury Notes (TY)
  • Lean Hogs (LH)
  • Australian Dollar (AD)
  • Heating Oil (HO)
  • Cotton (CT)
  • e-mini Nasdaq (NQ)

I picked these at random, one from each major market group. The test period will be from 1/1/2007 to 12/31/2011, a 5-year period that includes some quiet, and some very chaotic markets.

Finally, I will assume $5 round turn commissions, and $30 round turn slippage. The $30 slippage might be excessive, but I've always found it is better to be conservative than to underestimate slippage. I'd rather not be disappointed in real time with larger than expected slippage.

The System

I will use an extremely simple strategy for the tests I am running: a simple breakout based on closing prices. The system is always long or short, and will enter at the next bar open, long if the close is the highest close of the past X bars, and short if the close of the lowest close of the last X bars. One version of the system always exits at the end of the day, and the other version is a swing system. X will be varied from 5 bars to 100 bars.

Here is the code for the swing version, Strategy A:


Figure 1

Here is the code for the day trading version, Strategy B:

Figure 2

Figure 2

Day, or Night?

Since most markets these days run nearly 24 hours, we can run into issues when testing historically, as many markets traded "pit" hours years ago. Because of that, and because most of the volume is during these traditional pit hours, I will use the old pit session times, but still use electronic trading data.

Exit 1 Minute Before Close

If you use Tradestation to test, you may be familiar with their keyword "setexitonclose." This is a neat function for backtesting, but in live trading the order is sent after the market is closed, rendering it ineffective. So, I set up the custom sessions above to exit 1 minute before the old "pit" closing time. Then, I know my strategy will exit properly, and will therefore work in backtest and in real time.

Final Pre-Test Info

Now that we have everything properly setup, it is time to run some tests. For each instrument given above, I will run tests with 5, 15, 30, 60, 120, and Daily bars. That gives us 7 instruments x 6 bar sizes = 42 tests. Then, for each test, I will run the breakout length X from 5-100 in steps of 1, which yields 96 iterations per test. Since I am running two strategies overall, I am running 42 x 96 x 2 = 8,064 unique performance sets.

For each of the 42 tests for each strategy, I will record the best Net Profit (out of the 96 iterations) and its corresponding maximum drawdown (if best Net Profit is greater than zero). For each of the 42 tests, I will also record the percentage of iterations that are profitable.

All the tests being run are shown graphically in Figure 3.

Figure 3

Figure 3

Questions To Answer

Once all the tests are run, I will see if I can answer the three questions below. These questions are directly related to the desires of many discretionary day traders.

  1. Many traders love strategies that are the same across multiple instruments. An example of this is price action trading - many feel the principles of price action hold across all instruments. So, when these traders test algorithms, one demand is that a strategy is viable ONLY if it is profitable across many instruments. Question: Is multiple market profitability a reasonable requirement?
  2. Most day traders need to be out by the end of the day. Question: Does forcing an exit at the end of each day improve or decrease strategy performance?
  3. Many day traders try to go for the smallest (shortest) bar size possible. The theory is that trades will be more frequent, losses will be smaller, and exits will be more responsive to market conditions. Question: What kind of impact does bar size have on performance?

Before I get to the results, I should point out that this is one study, with two strategies, across only seven markets. So, the conclusions I reach may not hold up over all markets, and may be different for different strategies. I'm guessing, though, that the conclusions probably do hold in general.


Results of all 8,046 performance tests are shown in Table 1 for the swing Strategy A, and Table 2 for the intraday Strategy B. I will refer to these results as I discuss each of the three questions.

Table 1

Table 1


Table 2

Table 2

1. Is profitably across multiple instruments a reasonable requirement?

Figures 4 and 5 show the best case Net Profit for both the swing and intraday version of the strategy. With one notable exception (Heating Oil intraday trading), what is good in one market is generally good in another market. Although the maximum profit varies wildly from market to market, all the swing markets are profitable, and all but one of the intraday markets are unprofitable. This should give you some confidence that the strategy is sound (or not). Of course, it doesn't mean the strategy is tradable in each market, since I am looking only at maximum net profit based on optimization. But, clearly the intraday Strategy B is not viable in most markets.

Figure 4

Figure 4

Figure 5

Figure 5

So, the results show that profitably across multiple markets is possible, even with markedly different instrument, suggesting that is indeed a valid requirement.

2. Is exiting end of day (intraday trading) a good idea?

Figure 5 shows the answer loud and clear - the answer is no. You'll get more profit by swing trading. Unfortunately, it might also mean enduring more drawdowns, but think of it like this: the market "pays" people for holding overnight and weekends, and taking on that risk.

3. Are shorter timeframes better?

Since Strategy A is clearly the better strategy, I will use those results to look at the impact of bar length (timeframe). This is shown in Figure 6, where I look at the percent of iterations that were profitable for each combination of instrument and bar size. So, for example, Wheat with 5-minute bars shows that 62% of iterations were profitable (denoted with a red circle in Figure 6). This means that as I varied the breakout length from 5 to 100 in increments of 1 (96 total iterations), 60 of those runs resulted in a profitable end result. Obviously, the best case is 100% - where no matter what value you use for breakout length, the end result is positive.

Figure 6

Figure 6

Results of Figure 6 show that in general profitability increases as the bar length increases, although this effect is not very pronounced, and it does not hold for all instruments. This is confirmed by the results of Figure 4, where the maximum Net Profit generally increase with increasing bar period, but not dramatically.

Why is this so? Profitability could be better as bar period increases, since random market noise plays a smaller role in longer period bars (i.e., the true trend is more easily seen in larger period bars). Of course, drawdown should also be considered when looking at bar size, but in this study it does not seem to change greatly with bar size.


Results with this simple strategy lead to three conclusions:

  1. Striving for profit in multiple markets, as a confirmation of a strategy, is indeed possible.
  2. Swing trading is likely more profitable than intraday trading.
  3. Longer timeframes are generally superior to short time periods, but this is not a major effect.

Follow On

Of course, I made these conclusions based on one study. What if the strategy was different? What if the timeframes or markets were different? What if different years were used for the test period? Will the conclusions reached here still hold? I'll examine those questions in Part 2.

If you would like to learn more about building trading systems be sure to get a copy of my latest book, Building Winning Algorithmic Trading Systems.


Breakout Strategies (TradeStation ELD)
Breakout Strategies WorkSpace (TradeStation TWS)

-- Kevin J. Davey of KJ Trading Systems

About the Author Kevin Davey

Kevin Davey is a professional trader and a top performing systems developer. Kevin is the author of “Building Algorithmic Trading Systems: A Trader’s Journey From Data Mining to Monte Carlo Simulation to Live Trading” (Wiley Trading, 2014.) . He generated triple digit annual returns 148 percent, 107 percent, and 112 percent in three consecutive World Cup of Futures Trading Championships® using algorithmic trading systems.

His web site,, provides trading mentoring, trading signals, and free trading videos and articles. He writes extensively in industry publications such as Futures Magazine and Active Trader and was featured as a “Market Master” in the book The Universal Principles of Successful Trading by Brent Penfold (Wiley, 2010).
Active in social media, Kevin has over 15,000 Twitter followers. An aerospace engineer and MBA by background, he has been an independent trader for over 20 years. Kevin continues to trade full time and develop algorithmic trading strategies.

  • rob says:

    Thanks Kevin for the great article! I am looking for 5 to 10 non correlated futures instruments. Would the ones you used in your testing be a good basket for me to use or would you suggest some others?

  • Thanks for the kind comments, Rob. You can approach the non-correlation issue a few ways. You mention trading different instruments, which is very valid. You could also trade different timeframes, and different strategies altogether. Many times, just having a strategy “different” in 1 or 2 ways from the original strategy will give good diversification.

    No matter how you do it, you can measure diversification via a correlation analysis, or via a combining of equity curves. A smoother equity curve, for example, usually results from uncorrelated strategies.

    OK, but you asked about just uncorrelated markets. How would I do that? I’d look to grab 1 or 2 instruments from each of the major groups:

    stock indices
    interest rates
    agriculturals and meats

    If you do that, you should have a nice diversified basket to trade. Of course, there are times when everything becomes correlated (think 2008 Financial Crisis), but baskets are generally pretty good.


  • rob says:

    Thank you sir I really appreciate your response. By the way great book!

  • Thanks, Rob! I do appreciate it!!

  • Alex says:

    Any conclusions made from this article are plagued by data mining bias. In addition, the chosen time period suggests data snooping bias, meaning that a different time period could produce different results. Choice of instruments constitutes selection bias. Right there you have an article plagued by data mining, data snooping and selection bias. I wonder if the author is aware of these problems.

  • Hi Alex – Thanks for the comment.

    At the end of the article, I do state: “Of course, I made these conclusions based on one study. What if the strategy was different? What if the timeframes or markets were different? What if different years were used for the test period? Will the conclusions reached here still hold?” So, it is entirely possible that the conclusions reached will be different for different instruments and different time periods. I believe I was pretty upfront about that.

    Of course, one key thing to keep in mind is that I present results of maximum net profit quite a bit, and those kinds of results are very, very unlikely to be had in the future. They are optimized, after all. I should have made that clearer in the article. I think the more interesting point is that many times even the best case profit is downright terrible.

    Thanks again for taking time to respond.


  • Alex says:

    “So, it is entirely possible that the conclusions reached will be different for different instruments and different time periods. I believe I was pretty upfront about that.”

    It is certain that they will be different and since you admitted that up front, I wonder, what was the whole point of this exercise? To show what to whom exactly? I don’t get it. I spend time to read the article just to find out that no conclusion is certain from it. If that is the way to treat readers, well, you have accomplished your goal.

    I have read very good articles in this website by other authors from which I gained something. I don’t think this article deserved this place because its assumptions question its conclusions. We as traders have no time for that. This is noise, not a useful article. I am sorry but this is the truth.

    • Mark says:

      I totally disagree, Alex. The article shows one method of analysis you could take to reach the answers you seek. This could be replicated with other markets and other strategies. In case you are here to learn about trading system development, that could be very educational. If you already know all this stuff, Alex, then kick yourself for me because you’re here in the first place.

  • Marco says:

    Thanks for the great article! I think the questions that arise with it are many as you stated above, it can be used as a strong base to develop further the simple concept described.

  • I’m glad you found it useful, Marco!

  • Rich S. says:


    Being a little tough on Kevin aren’t ya? Granted, it was only over four years of data. However, it did demonstrate varying results with different time periods and holding times. It is up to you to explore further.


  • >