Fooled by Monte Carlo Analysis

Fool 2

–by  Michael Harris from Price Action Lab

Simple Monte Carlo analysis tools are often used to assess the risks of trading systems and to determine appropriate capitalization levels. However, simple trade reshuffling algorithms can produce misleading results in many cases and fool their users.

There are several Monte Carlo analysis tools available to traders. Some of these are even distributed for free via popular forums and blogs. Although these tools can be useful in certain cases, they can also produce misleading results when the following conditions are true:

  1. The systems are highly curve-fitted and/or have high win rate.
  2. There is forced long/short symmetry in trade generation.
  3. The systems operate in longer-term trend following.
  4. The systems take trades that depend on the outcome of certain other trades.
  5. The system trade generation uses equity curve feedback.
  6. The system dynamics depend on position sizing.
  7. The trade sample is quite large

Basic Monte Carlo Method

Given a vector of n trade results (P/L):

T[0] = {T1, T2, T3, …,Tn}

then k, new vectors may be generated by reshuffling the original trade results. There are now k+1 vectors of trade results, and corresponding equity curves, which can be used in accessing the potential of the system, its risk, drawdown, etc.

Rationale and assumptions

The rationale behind the method is that T[0] represents a particular order of trades and naturally the order could be different.  Reshuffling of results is employed to arrive at different possibilities. The main assumption here is that all outcomes in T[0] are independent in the same way that one coin toss is independent from another. If the independence condition is violated, then this particular method of assessing risk does not apply and may produce misleading results.

The independence assumption is directly violated in cases 2, 4, 5 and 6 above and indirectly but in a significant way in case 3. Below are a few examples for some of the cases, 1-7.

Case 1

Curve-fitted system with high win rate

The results of a long-only 50-200 golden cross system in S&P 500 are shown below:

 results of a long-only 50-200 golden cross system in S&P 500

This system has 100% win rate in SPY unadjusted since inception. This system may be considered curve-fitted. Monte Carlo will not work for this or for similar systems even with many more trades if they are curve- fitted because all new T[k] vectors generated by reshuffling will not deviate too far from curve-fitting conditions. This can also be proven mathematically.

Case 2

Forced long/short symmetry

Many trading systems employ forced long/short symmetry. Even systems with price patterns can have long/short symmetry as I have shown in another blog. Long/short symmetric systems have several advantages and some disadvantages, but an analysis along these lines is beyond the scope of the blog. The important point here is that most Monte Carlo algorithms break the symmetry that is present by design. As a result, any inferences made from them are false. Below is the equity curve for a 50-200 cross long/short symmetric system in SPY unadjusted data since inception:

 equity curve for a 50-200 cross long/short symmetric system in SPY

 

 

Monte Carlo analysis will break the long/short symmetry of this system and produce false results.

Case 3

Longer-term trend following

Actually, both systems in Cases 1 and 2 above were trend-following ones. If a system does longer-term trend following in daily or weekly timeframes, then the Monte Carlo method is likely not to be appropriate. Below is the equity curve of a system I use that emulates trend-following by generating long-only, short-term trades along trends. Naturally, it also generated many losing trades along downtrends.

 drawdown levels during the dot-com crash and financial crisis bear markets of 2000 and 2007

 

This system produced large drawdown levels during the dot-com crash and financial crisis bear markets of 2000 and 2007, as shown on the above chart. However, reshuffling the trades may and in many cases will produce a certain sequence T[k] that has losing trades from both downtrends. In essence, this would amount to expecting a 100% downtrend in SPY! This is unrealistic and the distribution of returns will cause a highly skewed median drawdown to the left. In turn, both risk of ruin and required initial capital may be overestimated, seriously limiting the profitability of the system in the future.

Cases 4, 5 and 6

The non-applicability of the simple Monte Carlo tools available to retail traders either at a price or for free arises in these cases from the fact that trades are not independent by design. Reshuffling of trades assumes independence and, as a result, any conclusions made do not conform to reality. Note that systems with trades that depend on the outcome of previous trades or on the rate of equity growth are common and actually have an edge in many cases. In addition, position size algorithms must be considered as an integral part of a trading system, otherwise the results may be misleading. It is quite interesting that some developers, or promoters, of the simplistic Monte Carlo approach argue that it must be used before applying position size. They argue that traders calculate capital requirements and risk of ruin without accounting for complete system dynamics. In this case, the results can greatly overestimate or underestimate these parameters.

Case 7

Large trade samples

In the presence of a large number of reshuffles k and when the number of trades is large, there is high portability of generating trade sequences T[k] with drawdown levels equal to the initial capital. Remember that the net profit of a system is equal to the sum of winners minus the sum of losers:

net profit = sum of winners – sum of losers

Next, consider a trading system with $100,000 initial capital but tested over a long period of time so that the sum of winners is $400,000 and the sum of losers is $200,000.  The profit factor of this system is 2 but the sum of losers is twice the initial capital. Considering trade sequences in the results with drawdown levels close or greater than $100,000 may be unrealistic. These sequences are outliers and often an artifact of the long backtest. In reality, they have probability near 0 but if one does many reshuffles a great number of such sequences will emerge due to the ordering of trades. In other words, the probability of all sequences of trades T[k] is not the same but it is generally a function of the path the system follows and of discrete time. This effect is not evident with a small number of trades but becomes dominant when the number increases.

Summary

Simple, Monte Carlo analysis tools can be useful in certain cases, provided that their users understand the limitation and assumptions. One can safely assume that in the case of more advanced trading systems that have an edge, these tools do not apply without major modifications or may not apply at all. They can serve as a training tool for aspiring system traders to understand the relationship between drawdown, initial capital, and risk of ruin but not for realistic calculations of these parameters except in some simple cases. Actually, methods for determining risk of ruin, initial capital, and maximum expected drawdown are quite involved and are considered an integral part of a trading edge by many quants. Simple reshuffling methods cannot fulfill this purpose due to the path dependence of return and risk, and also due to the dependence of return outcomes in many realistic cases.

–by  Michael Harris from Price Action Lab

About the Author Michael Harris

Michael Harris is a trading expert and a developer of advanced pattern recognition software for the benefit of position and swing traders. Michael developed APS Automatic Pattern Search software which has received great acclaim and recently Price Action Lab, a program that includes an advanced technical analysis indicator based on price patterns, called the p-Indicator. He also provides consulting services about trading system development and market analysis to institutional investors and hedge funds.In years past, Michael has also done work for a number of different financial firms, where he developed a bond portfolio optimization program and trading systems for commodities and stocks. Since 1989, he has been as an active trader.Michael is also a best selling author. His first book “Short-Term Trading with Price Patterns” was published in 1999. His other two books “Stock Trading Techniques with Price Patterns” and “Profitability and Systematic Trading” were published in 2000 and 2008, respectively.

follow me on:
  • David says:

    Thanks for this article. I also have reservations about the applicability of MC sim in some of the situations presented in the article. In my opinion the failure in the case of trend-following systems is a major drawback. This was well explained in the article. I think that MC sim may have good application in the case of scalping systems but never tried that.

    What do you think about the different method offered by Aronson in his book?

    Best.

    • Hello David,

      if you mean the MC method based on daily market returns I find it complicated to apply and it also suffers form similar issues. Thanks.

  • rob says:

    This is why you must perform detailed wfo testing before you run Monte Carlo testing.

  • Mark says:

    I think part of the rationale to leave position sizing out during testing is because a drawdown early is as bad as a drawdown late in the sequence from a MC standpoint. If you position size, though, then early drawdowns will be minimized since, with total equity not having had much time to grow yet, position size will be small. Where do you disagree?

    • The rationale is wrong because of several reasons that are beyond the scope of a short reply but primarily due to:

      1. You are assuming variable position risk
      2. You neglect friction effects

      No. 2 is even more important than the first in many cases. Many theoretical edges disappear when position size, starting capital and commissions are accounted for. I can recall the SPY overnight effect I have once talked about in my blog. The assumed “edge” disappears as soon as position size is included for small accounts with per share commission cost. Here are blogs about that:

      http://www.priceactionlab.com/Blog/2011/02/who-needs-the-trading-day/

      http://www.priceactionlab.com/Blog/2011/02/who-needs-the-trading-day-part-two/

      In general, trading system performance depends on:

      1. Initial capital (especially true for fx and futures)
      2. Commissions
      3. Position sizing
      4. Path followed

      Therefore, I fail to understand how some people can claim that an MC simulation without position sizing can generate realistic results. I guess this is not the only ludicrous claim around. There are others too.

      The idea is that MC simulation is not a realistic process. Therefore, your reversal of burden of proof does not apply, since as I have already argued one should not consider this method in the first place for reasons much more important than position sizing anyway.

      • Mark says:

        Hi Michael,

        I don’t understand what you mean by “variable position risk” or “friction effects.”

        I agree that trading system performance depends on path followed. However, if I understand correctly from a MC perspective, there’s no reason this path had to be the path we followed. Any influence, which is beyond definition or categorization [and therefore random for all practical purposes] could have led to an entirely different path. I am reminded of the ST:TNG episode “Parallels” (if you know of that) where an infinite possible number of realities appear at once.

        To me, this feels intuitively correct. It may not be, though… but how to prove that?

        If you’re not going to do MC analysis then what would you do instead?

        • Hi Mark, I think you are paying too much attention to No. 6, where I referred to cases where “The system dynamics depend on position sizing.”. Examples, are systems such as the SPY overnight effect, book scalping systems, etc. The other considerations are much more important and especially the fact that serial correlation is destroyed by MC, as well as long/short symmetry, etc. As far as to what I do instead, I have presented in my blog posts alternatives, such as for example tests on portfolios of securities and even strict validation on anti-correlated securities. Ex:http://systemtradersuccess.com/deterministic-machine-design-trading-systems-strict-cross-validation/

          No method works in all cases but MC is to be avoided in my opinion because it is a naive test.

          Best.

          • Mark says:

            Serial correlation is destroyed by MC: why? You’re taking the trades based on the same data series and randomizing the order. You are not randomizing the data series from which the trades were generated and coming up with new ones.

            Mark

  • Mark, when you randomize trades you are essentially randomizing the price series returns that correspond to those trades. The new random trade sequence could not have been generated from the original price series unless this is true. It is easy to show that: If your system rules are deterministic, then they will produce the same exact trade sequence each time they are backtested with the same price series. Only if you change the price series, then you can get a different trade sequence. If there is serial correlation in the returns and your system takes advantage of that, then randomizing the trade sequence destroys the serial correlation and the results of MC are bogus. You will need to show that the trading system rules do not take advantage of serial correlation by chance before you do the randomization. If you know a test for doing that please consider sending an article to Jeff for publication because that will be very interesting. Michael.

  • Mark says:

    Your first sentence makes sense.

    When I initially read your comment about serial correlation, it hit me with a thud. I still wonder whether that really matters? Serial correlation is the degree to which patterns in the past will predict future activity. It will therefore only be known in the future. It can’t be known right now.

    You could say “but if it has been working up until now, it will continue to work.” Surely, then, it’s something that will continue to have inertia until it doesn’t at which point you could suffer losses.

    I therefore don’t know why it should be brought into the discussion and why MC should not therefore be a routine part of the system development process.

    Systems become “broken” all the time. That’s the point at which serial correlation ends, right?

  • […] Simulationen scheitern nützliche Informationen zur Verfügung zu stellen (Ich schlage vor, this Artikel mehr über sie zu erfahren). Allerdings Monte Carlo Simulationen sind nicht nützlich […]

  • >