- You are here:
- Home »
- Blog »
- System Development
- » Fooled by Monte Carlo Analysis

–by Michael Harris from Price Action Lab

Simple Monte Carlo analysis tools are often used to assess the risks of trading systems and to determine appropriate capitalization levels. However, simple trade reshuffling algorithms can produce misleading results in many cases and fool their users.

There are several Monte Carlo analysis tools available to traders. Some of these are even distributed for free via popular forums and blogs. Although these tools can be useful in certain cases, they can also produce misleading results when the following conditions are true:

- The systems are highly curve-fitted and/or have high win rate.
- There is forced long/short symmetry in trade generation.
- The systems operate in longer-term trend following.
- The systems take trades that depend on the outcome of certain other trades.
- The system trade generation uses equity curve feedback.
- The system dynamics depend on position sizing.
- The trade sample is quite large

Given a vector of n trade results (P/L):

T[0] = {T1, T2, T3, …,Tn}

then k, new vectors may be generated by reshuffling the original trade results. There are now k+1 vectors of trade results, and corresponding equity curves, which can be used in accessing the potential of the system, its risk, drawdown, etc.

The rationale behind the method is that T[0] represents a particular order of trades and naturally the order could be different. Reshuffling of results is employed to arrive at different possibilities. The main assumption here is that all outcomes in T[0] are independent in the same way that one coin toss is independent from another. If the independence condition is violated, then this particular method of assessing risk does not apply and may produce misleading results.

The independence assumption is directly violated in cases 2, 4, 5 and 6 above and indirectly but in a significant way in case 3. Below are a few examples for some of the cases, 1-7.

Curve-fitted system with high win rate

The results of a long-only 50-200 golden cross system in S&P 500 are shown below:

This system has 100% win rate in SPY unadjusted since inception. This system may be considered curve-fitted. Monte Carlo will not work for this or for similar systems even with many more trades if they are curve- fitted because all new T[k] vectors generated by reshuffling will not deviate too far from curve-fitting conditions. This can also be proven mathematically.

Forced long/short symmetry

Many trading systems employ forced long/short symmetry. Even systems with price patterns can have long/short symmetry as I have shown in another blog. Long/short symmetric systems have several advantages and some disadvantages, but an analysis along these lines is beyond the scope of the blog. The important point here is that most Monte Carlo algorithms break the symmetry that is present by design. As a result, any inferences made from them are false. Below is the equity curve for a 50-200 cross long/short symmetric system in SPY unadjusted data since inception:

Monte Carlo analysis will break the long/short symmetry of this system and produce false results.

Longer-term trend following

Actually, both systems in Cases 1 and 2 above were trend-following ones. If a system does longer-term trend following in daily or weekly timeframes, then the Monte Carlo method is likely not to be appropriate. Below is the equity curve of a system I use that emulates trend-following by generating long-only, short-term trades along trends. Naturally, it also generated many losing trades along downtrends.

This system produced large drawdown levels during the dot-com crash and financial crisis bear markets of 2000 and 2007, as shown on the above chart. However, reshuffling the trades may and in many cases will produce a certain sequence T[k] that has losing trades from both downtrends. In essence, this would amount to expecting a 100% downtrend in SPY! This is unrealistic and the distribution of returns will cause a highly skewed median drawdown to the left. In turn, both risk of ruin and required initial capital may be overestimated, seriously limiting the profitability of the system in the future.

The non-applicability of the simple Monte Carlo tools available to retail traders either at a price or for free arises in these cases from the fact that trades are not independent by design. Reshuffling of trades assumes independence and, as a result, any conclusions made do not conform to reality. Note that systems with trades that depend on the outcome of previous trades or on the rate of equity growth are common and actually have an edge in many cases. In addition, position size algorithms must be considered as an integral part of a trading system, otherwise the results may be misleading. It is quite interesting that some developers, or promoters, of the simplistic Monte Carlo approach argue that it must be used before applying position size. They argue that traders calculate capital requirements and risk of ruin without accounting for complete system dynamics. In this case, the results can greatly overestimate or underestimate these parameters.

Large trade samples

In the presence of a large number of reshuffles k and when the number of trades is large, there is high portability of generating trade sequences T[k] with drawdown levels equal to the initial capital. Remember that the net profit of a system is equal to the sum of winners minus the sum of losers:

net profit = sum of winners – sum of losers

Next, consider a trading system with $100,000 initial capital but tested over a long period of time so that the sum of winners is $400,000 and the sum of losers is $200,000. The profit factor of this system is 2 but the sum of losers is twice the initial capital. Considering trade sequences in the results with drawdown levels close or greater than $100,000 may be unrealistic. These sequences are outliers and often an artifact of the long backtest. In reality, they have probability near 0 but if one does many reshuffles a great number of such sequences will emerge due to the ordering of trades. In other words, the probability of all sequences of trades T[k] is not the same but it is generally a function of the path the system follows and of discrete time. This effect is not evident with a small number of trades but becomes dominant when the number increases.

Simple, Monte Carlo analysis tools can be useful in certain cases, provided that their users understand the limitation and assumptions. One can safely assume that in the case of more advanced trading systems that have an edge, these tools do not apply without major modifications or may not apply at all. They can serve as a training tool for aspiring system traders to understand the relationship between drawdown, initial capital, and risk of ruin but not for realistic calculations of these parameters except in some simple cases. Actually, methods for determining risk of ruin, initial capital, and maximum expected drawdown are quite involved and are considered an integral part of a trading edge by many quants. Simple reshuffling methods cannot fulfill this purpose due to the path dependence of return and risk, and also due to the dependence of return outcomes in many realistic cases.

–by Michael Harris from Price Action Lab

Michael Harris is a trading expert and a developer of advanced pattern recognition software for the benefit of position and swing traders. Michael developed APS Automatic Pattern Search software which has received great acclaim and recently Price Action Lab, a program that includes an advanced technical analysis indicator based on price patterns, called the p-Indicator. He also provides consulting services about trading system development and market analysis to institutional investors and hedge funds. In years past, Michael has also done work for a number of different financial firms, where he developed a bond portfolio optimization program and trading systems for commodities and stocks. Since 1989, he has been as an active trader. Michael is also a best selling author. His first book “Short-Term Trading with Price Patterns” was published in 1999. His other two books “Stock Trading Techniques with Price Patterns” and “Profitability and Systematic Trading” were published in 2000 and 2008, respectively.