Is The MAC System Overly Optimized?

About the Author Jeff Swanson

Jeff is the founder of System Trader Success - a website and mission to empowering the retail trader with the proper knowledge and tools to become a profitable trader the world of quantitative/automated trading.

follow me on:
  • ross says:

    Jeff, when was trade #8? It looks like around half the total profit came from that one trade.

    • Ross, the largest trade was during the dot-com boom. That trade was from 1995-1998 when the S&P climbed from around 465 to 1020. This generated a profit of $118,610. The current open trade may be within the top three when it closes. The open profit is $44,900.

  • Alex says:

    Jeff, a curve-fitted system will be always robust with respect to parameter changes. Actually you proved that the system was curve-fitted not the opposite as you argue. You have to rethink your whole philosophy.The reason that the MAC system worked on SPX data is because there are no prolonged sideways periods like in some other markets. Actually the 1990s trend makes the system robust to paarmeter variations.

    • Hello Alex. I disagree that “a curve-fitted system will be always robust with respect to parameter changes”. I’ve seen overly optimized systems where you change a parameter and the equity curve falls apart. This would be a red flag in my book. With a system that shows robustness across many different parameters, I feel much better. Besides, curve-fitting is not the problem. Every system is curve-fitted to the historical data. I rely on historical patterns or behavior repeating into the future. Of course if the market changes dramatically (stops trending for a prolonged period), the system will stop working. This is nothing new as the MAC system is a trend following system. Of course a trend following system will stop working when a market no longer trends. Discretionary traders as well as system traders adapt over time as edges appear and disappear. Knowing when to stop trading a system is part of any trading plan. Some edges last decades, others don’t. For the MAC system you can remove the 1990’s trend and it still works and the parameters remain robust. The edge exploited by the MAC system may last for another 100 years, or it may die over the next couple of years. No one knows. All I can do is make best estimates based upon history and then remain flexible at what the future holds. If one had to wait for 100% certainty no trades would be taken.

  • Marco says:

    Despite everything I believe it would be really interesting to see how the system behaves with RUSSEL 2000 or NASDAQ, just to see if we can boost returns more.
    https://nightlypatterns.wordpress.com

    • Hello Marco. I just ran the system on SPY, ES, QQQ, DOW, and IWM. All produce positive results. For fun I ran it on APPL, GLD and USO. Again, positive results except for USO. While the USO equity curve is still in the green, it’s an ugly falling equity curve. It’s clear the system was designed for the S&P, but it will hold-up on other markets.

  • Alex says:

    “Besides, curve-fitting is not the problem”

    Curve-fitting IS the problem. There’s NO edge to the MAC system. It’s all an illusion you have fallen victim of.

    Jeff, curve-fitting and optimization are not equivalent notions. You seem to use them interchangeably. The formal definition of optimization is that it is a global maximum that is robust when parameters are perturbed. What you have seen that broke down was an unstable local maximum. The MAC system parameters have been selected for a global maximum for SPX. Why don’t you apply it to some commodity futures?

    Jeff the MAC system has 5 parameters. Any claim that it is not fitted is ludicrous to say the least. BTW just trying to help as I have fallen a victim of such systems in the past and have lost money.

    • Hello Alex. I appreciate your opinion and desire to help. Thanks. However, I do disagree. In my opinion curve-fitting is not the problem as all systems are curve fitted to the historical data. Over-optimization would be the potential problem. So, yes curve-fitting and over optimization are two different aspects which was one of the points of the article. As you state, selecting local maximum values for each individual input parameter would likely increase the chances for having an over-optimized system and this would be a problem. As stated in the article, I found the system not to be overly-optimized. Could the system be overly optimized? Maybe, but I did not find evidence to support this. Might the system not work into the future? Maybe. If the market drastically deviates from it’s historical behavior, it will stop working. Furthermore, it has been my experience that systems don’t need to work across different markets. Ideally they should work on similar markets. In this case, the system was built for the stock index markets. Specifically the S&P cash market. However it should fair well on the S&P futures, the DOW, QQQQ and others of that nature. Developing the system across similar (highly correlated) markets would be another method to avoid over-optimization as would using in-Sample/Out-of-Sample data during the development period. In the end, if the system does not do well on Gold, Oil or Apple stock, that would not necessarily be a “bad” thing. I’ve seen far too many systems work for years and years when build for specific markets. I just ran the system on SPY, ES, QQQ, DOW, and IWM. All produce positive results. For fun I ran it on APPL, GLD and USO. Again, positive results except for USO. While the USO equity curve is still in the green, it’s an ugly falling equity curve. It’s clear the system was designed for the S&P, but it will hold-up on other markets. This is another bit of evidence that the system is not overly-optimized to the historical data.

  • Alex says:

    Some more comments:

    I get 27 trades in the same period in SPX with data from Yahoo finance and a backtest in excel. Would it be possible to reply with your EL code in text form since I have no TS? Do you check for a cross before you enter and exit? Any additional rules other than those stated?

    Then I suggest the following:

    Please report the max % DD, not $$.

    Try SPY because in reality you cannot buy SPX shares and the backtest is unrealistic.

    What are the units of expectancy and what it stands for? Is it average trade divided by average loser?

    What is the Sharpe ratio? I get about 0.60. This should tell you whether this is just a fluke or not. At 0.60 it is a fluke. You should have more than 1.50 to be sure.

    • Alex says:

      Correction: It is 21 trades I get, not 27 trades.

    • Alex, I posted some results at the bottom of the article. As for your questions, there is no check for a cross on a buy. The 34-EMA must be 1.001 times the value of the 200-SMA before a long will be entered. When a 40-SMA crosses below the 200-SMA the position is closed. Hope this helps.

      • Alex says:

        Hi Jeff,

        “The 34-EMA must be 1.001 times the value of the 200-SMA before a long will be entered.”

        But in the article you say:

        “The 34-day exponential moving average (EMA) of the S&P 500 becomes greater than 1.001 times the 200-day EMA.”

        SMA or EMA?

        Then, you say

        “The P&L is not accumulated”

        What do you mean here? The CAR you got cannot be true unless you fully reinvented the closed equity on each trade.

        Then, here you say:

        “Dates tested are from October 1969 through September 2014”

        But below the equity graph the starting date is shown as 10/11/1979

        As you see, besides the curtve-fitting issues, there are some other issues with the article. Regardless, what you have proved is that this is a very robust curve-fitted system on SPX data. It will work only on price series with strong uptrends followed by downtrends. If there is no uptrend for a few years but only sideways action, those who use this system will go bust big way.

        • Alex, Sorry for the confusion on the rules. Please check the article for the entry/exit rules. They should be stated correctly. Likewise, the annualized return was wrong along with the equity curve was not correct. Those images were from a different MAC-System run run I was testing. Thanks for pointing that out.
          I almost did not even bother to include a performance report and equity graph in this article because that’s not the point of the article. The point was to demonstrate a technique on testing the robustness of input values. A technique anyone could apply to a system to help gauge if it’s overly optimized.

          “The P&L is not accumulated” means profits are not added to the trading equity. Instead, each trade is simply based upon the starting equity.

          I agree that this is a trend following system. Thus, it requires a trending market. This is really no different than nearly all trading systems I’ve ever seen. They work because they take advantage of historical market characteristics. When those historical characteristic fade and die, so does the system. This is nothing new and something we all live with as a traders. In short, you can always find an excuse no matter how well built and tested a trading system is, on why you should not trade it.

  • Alex says:

    Thanks Jeff but please note that although the number of trades was matched after the clarification you made, the net profit does not because you number of shares rules seems not to make any sense. The position sizing formula you suggest may generate more shares that the available equity depending on the value of ATR(10).

    Shares = $5,000 per trade / 5 * ATR(10)

    Actually you are not “always trading a small percentage of our starting capital” with this formula. If you want to do that you should select the amount and divide by the price to open the position.

    Unfortunately, your results cannot be reproduced and that also casts doubts on your claims that this is not an overly optimized system. My analysis shows that this is actually a highly optimized system to follow steep trends. It would be good if you posted your EL code in test form.

    More importantly, the procedure you followed actually tests if a system curve-fit is robust, not if a system is overly optimized.

    • Hello Alex, the formula is actually a function and will not allow more shares to be purchased than the given account size. Please note, the results on this page are not the focus here. This is not the MAC-System as defined by the author of the original system and defeats the purpose of this article which is testing of the input parameters – not the performance of the trading model. I would urge you to contact the original creator of the MAC-System to discuss exact backtesting and historical performance. I disagree that this system is overly optimized. I also understand that you feel it is overly optimized. Maybe you can post your results on why you feel it’s overly optimized?

      • Alex says:

        Jeff, thanks, Your articles are good to stimulate thinking but I think many will agree with me on this one. I wrote:

        “…the procedure you followed actually tests if a system curve-fit is robust, not if a system is overly optimized.”

        “Optimized” is a vague term. With systems we are interested to know whether there is curve-fit and how strong it is. You proved beyond doubt the opposite of what you think in my opinion, i.e. that the system is well curve-fitted because it responds well to parameters changes. You think that it means it is not overly optimized because of that. But this is a distracting issue in a way. Think about it.

  • >