Finding Out What Works, And What Doesn’t Work – Part II

In Part 1 of this series, found here, I ran a simple study of a trading system, in order to look at some of the common components (such as end of day closes, short versus long bar periods) that many system developers add to their trading system. Could some of these strategy building blocks be hurting their development efforts? To answer the question, I analyzed one trading system over 7 different markets for a 5 year period. Even though I was able to extract some conclusions from the study, I also said the following:

“Of course, I made these conclusions based on one study. What if the strategy was different? What if the timeframes or markets were different? What if different years were used for the test period? Will the conclusions reached here still hold? I’ll examine those questions in Part 2.”

To summarize the earlier work, the conclusions from Part 1 were:

  1. Striving for profit in multiple markets, as a confirmation of a strategy, is indeed possible.
  2. Swing trading is likely more profitable than intraday trading.
  3. Longer timeframes are generally superior to short time periods, but this is not a major effect.

In this second part of the study, I’ll now look at a different group of markets, a completely different strategy and some different timeframes. I’ll also try to answer the following questions: Will the conclusions still hold up? What type of insights, if any, do conducting studies like these actually provide? Finally, can studies like these improve your own strategy development?

Ground Rules

Previously in Part 1, I tested 7 different markets across a range of commodities.

(Group 1):

  • Wheat (W)
  • 10 Year Treasury Notes (TY)
  • Lean Hogs (LH)
  • Australian Dollar (AD)
  • Heating Oil (HO)
  • Cotton (CT)
  • the e-mini Nasdaq (NQ)

I picked these at random, one from each major market group. The test period will be from 1/1/2007 to 12/31/2011, a 5 year period that includes some quiet, and some very chaotic, markets.

For Part 2, I will test another basket of commodities.

(Group 2):

  • Live Cattle (LC)
  • Canadian Dollar (CD)
  • Copper (HG)
  • Natural Gas (NG)
  • Sugar (SB)

I will also test the period 1/1/2012 to present, a smaller test period outside of the range previously tested.

Finally, I will assume $5 round turn commissions and $30 round turn slippage.

The Systems

I will use two extremely simple base strategies for the tests I am running: a simple breakout based on closing prices (Strategies A and B), or a reverse breakout (countertrend – Strategies C and D). In all cases, the breakout length X will be varied from 5 bars to 100 bars.

Here is the code for all four versions of the strategy:

Results – Repeat of Part 1 Test, Different Instruments

First, I will conduct a test of Strategies A and B for the same period, with the second group of commodities. To keep things manageable, only the group summary results are shown below:

Strategy A- Breakout, Swing Trading

$ Thousands ($K) Maximum Net Profit, 2007-2011

5 min

15 min

30 min

60 min

120 min

daily

Median Group 1

$32

$42

$37

$38

$38

$39

Median Group  2

$24

$25

$27

$29

$37

$38

$ Thousands ($K) Maximum Drawdown for Max Net Profit case, 2007-2011

5 min

15 min

30 min

60 min

120 min

daily

Median Group 1

$16

$19

$12

$16

$19

$14

Median Group  2

$27

$45

$33

$10

$15

$6

% of Intereaction Profitable (100%=goal) 2007-2011

5 min

15 min

30 min

60 min

120 min

daily

Median Group 1

48.5

81

62.5

77

74.5

88

Median Group  2

24.5

18

25

30

69

60

Strategy B- Breakout, Exit of the day

$ Thousands ($K) Maximum Net Profit, 2007-2011

5 min

15 min

30 min

60 min

120 min

daily

Median Group 1

$19

$14

$13

$8

$5

$4

Median Group  2

$27

$19

$21

$18

$12

$2

$ Thousands ($K) Maximum Drawdown for Max Net Profit case, 2007-2011


no data shown for cases where maximum net profit <$0

5 min

15 min

30 min

60 min

120 min

daily

Median Group 1

$27

$19

$21

$18

$12

$2

Median Group  2

$19

$26

$10

$8

$8

% of Iterations Profitable (100%=goal), 2007-2011


no data shown for cases where maximum net profit <$0

5 min

15 min

30 min

60 min

120 min

daily

Median Group 1

58

90

97

62.5

100

98

Median Group  2


49

77

80


The results of Group 2 are generally in line with Group 1 results, which show that swing trading (strategy A) is better than intraday trading (Strategy B), and that longer timeframes are slightly better than short timeframe bars.

Results–Different Timeframes, Different Time Period, Different Strategy

Of course, concluding that long timeframe bars and swing trading is superior based on only a few tests is dangerous. So, let’s look at Strategies C & D (the opposite Strategies of A & B), and see if the results have the same tendencies for different timeframes and test period, on a random subset of Group 1 and Group 2 instruments.

Strategy C- Reverse Breakout, Swing Trading

$ Thousands ($K) Maximum Net Profit, 2012 - present

10 min

45 min

90 min

W

$17

$19

$17

AD

$36

$27

$23

HG

$73

$67

$68

SB

$6

$1

$3

Median-Subset Group

$27

$23

$20

% of Iterations Profitable (100%=goal), 2012 - present

10 min

45 min

90 min

W

50%

70%

46%

AD

88%

61

36%

HG

83%

100%

80%

SB

0%

10%

22%

Median-Subset Group

67%

66%

41%

Strategy D- Reverse Breakout, Exit End of the day

$ Thousands ($K) Maximum Net Profit, 2012 - present

10 min

45 min

90 min

W

$26

$11

$10

AD

$1

$1

$2

HG

$10

$9

$4

SB

$19

$12

$8

Median-Subset Group

$15

$10

$6

% of Iterations Profitable (100%=goal), 2012 - present

10 min

45 min

90 min

W

0

0

0

AD

0

0

0

HG

0

0

0

SB

0

0

0

Median-Subset Group

0

0

0

For timeframes, the results show a tendency for the smaller timeframes to be better than the longer timeframes, exactly the opposite of what was found earlier. But this effect is small, which could mean it is not actually true, and could be just an artifact of the limited testing. This is a good example why it is dangerous to draw conclusions from only a few tests.

For the swing vs. intraday testing, the results are pretty clear – intraday testing is not nearly as good as swing trading. This agrees with the earlier conclusion.

Conclusion

Through both Part 1 and Part 2 of this study, a lot of tests have been run over various bar timeframes, instruments, strategies and time periods. Yet, it would be dangerous to make any sweeping conclusions based on these tests. Although these results show that intraday trading is less profitable than swing trading, there are surely cases where intraday trading is preferable over swing trading.

So, what can this study actually teach us? I think the following lessons come out of this study:

  1. Intraday trading is less profitable than swing trading. This does not mean this always holds, but if you do most of your testing intraday and you are struggling to develop a strategy, perhaps opening up the strategy to allow swing trading is a good idea. It could be that you have a good swing strategy, but the end of day exit alone makes the strategy look bad.
  2. Longer timeframes are better (Part 1 conclusion), except when they aren’t (Part 2 conclusion). So, be careful of making conclusions when the effect is small, or results are contradictory. Since many strategy developers tend to find something that does or does not work and then focus only on that (or avoid it), the lesson here is that the developer should not “box” themselves in. Maybe short timeframes rarely work with your strategies, but that does not mean you should never test them. There will be other strategies where shorter time bars are an asset.
  3. Since a developer can never test everything, it may be worthwhile to vary your test approach. By looking at different timeframes, instruments, etc., you may discover new paths to take in your strategy development.

The key to successful strategy testing and development is to keep an open mind, and allow the data to point you in the right direction. At the same time, though, don’t let the data trick you into making false conclusions or sweeping generalizations. Remaining flexible in your development approach can lead to new and profitable strategies – ones that you might now be dismissing based solely on past testing.

If you would like to learn more about building trading systems be sure to get a copy of my latest book, Building Winning Algorithmic Trading Systems.

Download

Breakout Strategies (TradeStation ELD)
Breakout Strategies WorkSpace (TradeStation TWS)

— Kevin J. Davey of KJ Trading Systems

If you want to learn more about algo trading get a copy of my book,
"Introduction To Algo Trading"

Kevin Davey Introduction to Algo Trading

About the Author Kevin Davey

Kevin Davey is a professional trader and a top performing systems developer. Kevin is the author of “Building Algorithmic Trading Systems: A Trader's Journey From Data Mining to Monte Carlo Simulation to Live Trading” (Wiley Trading, 2014.) . He generated triple digit annual returns 148 percent, 107 percent, and 112 percent in three consecutive World Cup of Futures Trading Championships® using algorithmic trading systems.His web site, www.kjtradingsystems.com, provides trading mentoring, trading signals, and free trading videos and articles. He writes extensively in industry publications such as Futures Magazine and Active Trader and was featured as a “Market Master” in the book The Universal Principles of Successful Trading by Brent Penfold (Wiley, 2010). Active in social media, Kevin has over 15,000 Twitter followers. An aerospace engineer and MBA by background, he has been an independent trader for over 20 years. Kevin continues to trade full time and develop algorithmic trading strategies.

  • Marco says:

    I believe everytrader to think about intraday trading with high leverage. But as you shown with your results it’s not so easy like in swing trading. I really would like to come through something to add to my overnight trading. Have you got some ideas in which direction focus my research?
    https://nightlypatterns.wordpress.com

    • Thanks for the comment. Overnight trading, by itself, has always been challenging, at least for me. I find the best results from trading both day and night together.

  • Alex says:

    “The results of Group 2 are generally in line with Group 1 results, which show that swing trading (strategy A) is better than intraday trading (Strategy B), and that longer timeframes are slightly better than short timeframe bars.”

    That is not a valid conclusion since you have not normalized the results. You are comparing apples and oranges here.

    • Thanks for the comment, Alex. I am not certain I understand you. Can you please provide an example? Thanks!

  • Jim Vogt says:

    Kevin,

    Thank you for publishing this research. I think the results are very helpful.

    You tested your data with a very wide range of X values (breakout length) from 5 to 100. You have not reported on which were most profitable. Did your research indicate that longer or shorter breakout lengths were preferable?

  • Hi Jim –

    Thanks for the comment. To answer your question, I did not really examine which breakout length was best. The main reason is because the right length to use would be determined later in the process, when I did walkforward testing. I’ll probably address this in a future article.

    Thanks!

  • >