Share It!

# Fooled by Monte Carlo Analysis

–by  Michael Harris from Price Action Lab

Simple Monte Carlo analysis tools are often used to assess the risks of trading systems and to determine appropriate capitalization levels. However, simple trade reshuffling algorithms can produce misleading results in many cases and fool their users.

There are several Monte Carlo analysis tools available to traders. Some of these are even distributed for free via popular forums and blogs. Although these tools can be useful in certain cases, they can also produce misleading results when the following conditions are true:

1. The systems are highly curve-fitted and/or have high win rate.
2. There is forced long/short symmetry in trade generation.
3. The systems operate in longer-term trend following.
4. The systems take trades that depend on the outcome of certain other trades.
5. The system trade generation uses equity curve feedback.
6. The system dynamics depend on position sizing.
7. The trade sample is quite large

### Basic Monte Carlo Method

Given a vector of n trade results (P/L):

T[0] = {T1, T2, T3, …,Tn}

then k, new vectors may be generated by reshuffling the original trade results. There are now k+1 vectors of trade results, and corresponding equity curves, which can be used in accessing the potential of the system, its risk, drawdown, etc.

### Rationale and assumptions

The rationale behind the method is that T[0] represents a particular order of trades and naturally the order could be different.  Reshuffling of results is employed to arrive at different possibilities. The main assumption here is that all outcomes in T[0] are independent in the same way that one coin toss is independent from another. If the independence condition is violated, then this particular method of assessing risk does not apply and may produce misleading results.

The independence assumption is directly violated in cases 2, 4, 5 and 6 above and indirectly but in a significant way in case 3. Below are a few examples for some of the cases, 1-7.

### Case 1

Curve-fitted system with high win rate

The results of a long-only 50-200 golden cross system in S&P 500 are shown below:

This system has 100% win rate in SPY unadjusted since inception. This system may be considered curve-fitted. Monte Carlo will not work for this or for similar systems even with many more trades if they are curve- fitted because all new T[k] vectors generated by reshuffling will not deviate too far from curve-fitting conditions. This can also be proven mathematically.

### Case 2

Forced long/short symmetry

Many trading systems employ forced long/short symmetry. Even systems with price patterns can have long/short symmetry as I have shown in another blog. Long/short symmetric systems have several advantages and some disadvantages, but an analysis along these lines is beyond the scope of the blog. The important point here is that most Monte Carlo algorithms break the symmetry that is present by design. As a result, any inferences made from them are false. Below is the equity curve for a 50-200 cross long/short symmetric system in SPY unadjusted data since inception:

Monte Carlo analysis will break the long/short symmetry of this system and produce false results.

### Case 3

Longer-term trend following

Actually, both systems in Cases 1 and 2 above were trend-following ones. If a system does longer-term trend following in daily or weekly timeframes, then the Monte Carlo method is likely not to be appropriate. Below is the equity curve of a system I use that emulates trend-following by generating long-only, short-term trades along trends. Naturally, it also generated many losing trades along downtrends.

This system produced large drawdown levels during the dot-com crash and financial crisis bear markets of 2000 and 2007, as shown on the above chart. However, reshuffling the trades may and in many cases will produce a certain sequence T[k] that has losing trades from both downtrends. In essence, this would amount to expecting a 100% downtrend in SPY! This is unrealistic and the distribution of returns will cause a highly skewed median drawdown to the left. In turn, both risk of ruin and required initial capital may be overestimated, seriously limiting the profitability of the system in the future.

### Case 7

In the presence of a large number of reshuffles k and when the number of trades is large, there is high portability of generating trade sequences T[k] with drawdown levels equal to the initial capital. Remember that the net profit of a system is equal to the sum of winners minus the sum of losers:

net profit = sum of winners – sum of losers

Next, consider a trading system with \$100,000 initial capital but tested over a long period of time so that the sum of winners is \$400,000 and the sum of losers is \$200,000.  The profit factor of this system is 2 but the sum of losers is twice the initial capital. Considering trade sequences in the results with drawdown levels close or greater than \$100,000 may be unrealistic. These sequences are outliers and often an artifact of the long backtest. In reality, they have probability near 0 but if one does many reshuffles a great number of such sequences will emerge due to the ordering of trades. In other words, the probability of all sequences of trades T[k] is not the same but it is generally a function of the path the system follows and of discrete time. This effect is not evident with a small number of trades but becomes dominant when the number increases.

### Summary

Simple, Monte Carlo analysis tools can be useful in certain cases, provided that their users understand the limitation and assumptions. One can safely assume that in the case of more advanced trading systems that have an edge, these tools do not apply without major modifications or may not apply at all. They can serve as a training tool for aspiring system traders to understand the relationship between drawdown, initial capital, and risk of ruin but not for realistic calculations of these parameters except in some simple cases. Actually, methods for determining risk of ruin, initial capital, and maximum expected drawdown are quite involved and are considered an integral part of a trading edge by many quants. Simple reshuffling methods cannot fulfill this purpose due to the path dependence of return and risk, and also due to the dependence of return outcomes in many realistic cases.

–by  Michael Harris from Price Action Lab

### About the Author Michael Harris

• David says:

Thanks for this article. I also have reservations about the applicability of MC sim in some of the situations presented in the article. In my opinion the failure in the case of trend-following systems is a major drawback. This was well explained in the article. I think that MC sim may have good application in the case of scalping systems but never tried that.

What do you think about the different method offered by Aronson in his book?

Best.

• Hello David,

if you mean the MC method based on daily market returns I find it complicated to apply and it also suffers form similar issues. Thanks.

• rob says:

This is why you must perform detailed wfo testing before you run Monte Carlo testing.

• Mark says:

I think part of the rationale to leave position sizing out during testing is because a drawdown early is as bad as a drawdown late in the sequence from a MC standpoint. If you position size, though, then early drawdowns will be minimized since, with total equity not having had much time to grow yet, position size will be small. Where do you disagree?

• The rationale is wrong because of several reasons that are beyond the scope of a short reply but primarily due to:

1. You are assuming variable position risk
2. You neglect friction effects

No. 2 is even more important than the first in many cases. Many theoretical edges disappear when position size, starting capital and commissions are accounted for. I can recall the SPY overnight effect I have once talked about in my blog. The assumed “edge” disappears as soon as position size is included for small accounts with per share commission cost. Here are blogs about that:

In general, trading system performance depends on:

1. Initial capital (especially true for fx and futures)
2. Commissions
3. Position sizing
4. Path followed

Therefore, I fail to understand how some people can claim that an MC simulation without position sizing can generate realistic results. I guess this is not the only ludicrous claim around. There are others too.

The idea is that MC simulation is not a realistic process. Therefore, your reversal of burden of proof does not apply, since as I have already argued one should not consider this method in the first place for reasons much more important than position sizing anyway.

• Mark says:

Hi Michael,

I don’t understand what you mean by “variable position risk” or “friction effects.”

I agree that trading system performance depends on path followed. However, if I understand correctly from a MC perspective, there’s no reason this path had to be the path we followed. Any influence, which is beyond definition or categorization [and therefore random for all practical purposes] could have led to an entirely different path. I am reminded of the ST:TNG episode “Parallels” (if you know of that) where an infinite possible number of realities appear at once.

To me, this feels intuitively correct. It may not be, though… but how to prove that?

If you’re not going to do MC analysis then what would you do instead?

• Hi Mark, I think you are paying too much attention to No. 6, where I referred to cases where “The system dynamics depend on position sizing.”. Examples, are systems such as the SPY overnight effect, book scalping systems, etc. The other considerations are much more important and especially the fact that serial correlation is destroyed by MC, as well as long/short symmetry, etc. As far as to what I do instead, I have presented in my blog posts alternatives, such as for example tests on portfolios of securities and even strict validation on anti-correlated securities. Ex:http://systemtradersuccess.com/deterministic-machine-design-trading-systems-strict-cross-validation/

No method works in all cases but MC is to be avoided in my opinion because it is a naive test.

Best.

• Mark says:

Serial correlation is destroyed by MC: why? You’re taking the trades based on the same data series and randomizing the order. You are not randomizing the data series from which the trades were generated and coming up with new ones.

Mark

• Mark says: