Broken Strategy or Market Change: Investigating Underperformance

Broken Strategy

I recently had someone email me about the performance of a strategy I created back in late 2005/early 2006 and traded for a few years. I remember the strategy being a daily mean reversion set up with an intraday pullback entry. I figured it probably had not done well over the last decade. I stopped trading in the middle of 2008 because I did not like how it was behaving. In the backtest it did well in bear markets but was not doing so in the middle of 2008.

I ran the strategy from 2007, using the rules as they were published and was pleasantly surprised by the results. A CAR of 25%. Overall not too bad. Wish I had still been trading it. This is an ​11-year out of sample test.

Then I remembered, didn’t this have insane results from 2000 to 2005? Here are those results.

Did you notice that 595% return in 2003? Now I am thinking what the heck happened since 2007. Look at the drop in yearly returns. Was the original strategy overfit and flawed? Did the markets change? If so, what is the difference? What follows is my investigation into trying to determine if the strategy is broken or the markets have changed.

Rules

I cannot share the rules because of the NDA I signed when it was created. What I can say is that it is a very typical mean reversion strategy that trades stocks.

I will include 2006 in the In-sample period because the return for the year was in line with previous years.

Curve Fit?

Was the initial strategy curve fit? I do not have the original code so I made a best guess of the parameters I would have optimized on and their values. The strategy CAR was 1.3 standard deviations from the average of all the variations. Even the worst variation still had a very good CAR from 2000 to 2005 and saw similar drop in performance. This variation may have been a little overfit but that is not the entire story.

Number of Trades

Is there a decline in the number of trades? Average number of trades per year:

  • 2000 to 2006: 171
  • 2007 to 2016: 88

Wow, a drop of 50%! That starts to explain a lot. But why less trades?

Trading Universe

Are fewer stocks passing our liquidity filter? This is the average per year:

  • 2000 to 2006: 978,239
  • 2007 to 2016: 861,763

A small decline but it does not explain the large drop.

Number of Setups

Since the strategy depends on intraday sell off, maybe we are seeing less setups. This is the average per year:

  • 2000 to 2006: 5307
  • 2007 to 2016: 2537

That explains the lower number of trades per year. We have less setups. This means we are seeing fewer stocks sell off and setting up for a mean reversion trade. This explains a lot. But let us see if it is the entire story.

Avg % P/L per Trade

Is the quality of the trade dropping too? Meaning what is the average % profit/loss per trade?

  • 2000 to 2006: 6.25%
  • 2007 to 2016: 2.73%

Another drop of 50%. A double whammy. These last two stats explain the drop in the performance. But I wanted to know more.

Volatility

Are the trades less volatile and is that why they are not as big? These are the average 100-day historical volatility of the trades:

  • 2000 to 2006: 116%
  • 2007 to 2016: 106%

That is not leading to the decrease. What is causing the decrease in average % p/l? I don’t know what else to look at.

Final Thoughts

Is this strategy broken? I don’t think so. But what has changed is the market, which has greatly reduced the returns. Having fewer setups and smaller gains on the trades seems to have caused the reduction in returns. As to why, my theory is that it’s because of the popularity of mean reversion strategies and quantified trading. Do you have any ideas on what is going on or other tests you want to see? Post them in the comments below.

Remember we had eleven years of out-of-sample data to work with to determine if the strategy was originally overfit or if it had broken, or if the markets had changed. Even with all these data it was not completely easy to figure out what was going on. The next time your strategy starts to perform poorly after 3 to 12 months, remember how hard it can be. I dropped the strategy after 6 months. It was not broken, the markets had changed.

--by Cesar Alvarez from Blog Alvarezquanttrading

Good quant trading,

About the Author System Trader Success Contributor

Contributing authors are active participants in the financial markets and fully engrossed in technical or quantitative analysis. They desire to share their stories, insights and discovers on System Trader Success and hope to make you a better system trader. Contact us if you would like to be a contributing author and share your message with the world.

  • Carlos says:

    I am trading a similar strategy and have the same observations. In my backtest I also see fantastic returns before 2006 and good returns afterwards. Something major happened in the market around that time that got most of the edge of the strategy arbitraged away. Maybe high frequency trading?. But it’s still a great strategy when run against Russell 3000 stocks.

    I notice the majority of the trades are in biotech and technology. Not surprising as these are the most volatile stocks. These same stocks give poor return in my trend following system. So it serves as good diversification. Trend following mainly with less volatile stocks, and pullback mean reversion for the most volatile ones.

  • Ola says:

    Hi Cesar, thanks for posting. This confirms my own experience of mean reversion trading on the US market. The equity curve has been falling off:
    2010 35.9%
    2011 31.7%
    2012 20.3%
    2013 29.8%
    2014 10.5%
    2015 15.0%
    2016 7.2%

  • Very good article and thank you for sharing.

    Here are my thoughts, I am just beginner system developer.

    1. During in sample testing, the system had triple digits returns year after year, during the first and even 2nd year of live running the system had very low returns. In 2008 it had negative returns. So the system was heavily curved fitted. I highly doubt everyone started running mean reversion strategies in 2007 the first year the system went live or 2008. Look at 2006, it had 174% return and in 2007 only 21% return. That simply means whatever happened in back test was curve fitted too much.

    Also, maybe the system was tested on bad data too.

    If I build a system and it had triple digits in training, the very next year should be similar or something in backtest is wrong.

  • >