Using System Parameter Randomization To Estimate Future Returns

About the Author Jeff Swanson

Jeff is the founder of System Trader Success - a website and mission to empowering the retail trader with the proper knowledge and tools to become a profitable trader the world of quantitative/automated trading.

follow me on:
  • […] Using System Parameter Randomization To Estimate Future Returns [System Trader Success] You just spent a ton of time creating a trading system and being very careful not to over-optimize. You then tested it on the out-of-sample data segment and the performance looks good. What's next? Jump right into the live market? Maybe. But instead, you would like to perform one more test called System Parameter Randomization. The article, System Parameter Permutation a better […]

  • Hi Jeff, thanks for posting this and especially the video. It is very instructive on the mechanics of SPR in TradeStation. However I do want to caution readers that SPR is a tool and as such can be misused resulting in unrealistic expectations. A few caveats:

    1) Only 10 years of price data history was used. This system uses the E-mini so all available price history should be used. I’m not sure how far back TradeStation data goes but I’m sure it is more than 10 years.
    2) Only two parameters are varied in this example. All parameters should be randomized in SPR, not just some subset. The only exception would be the position size parameter. Like I said in the last blog post on this topic: just evaluating a single indicator (or a couple) does not enable enough random interactions. SPR is a method intended for application to a complete system in a portfolio context, meaning inclusion of filters, setups, entries, exits, signal ranking rules, and money management rules and also including commissions and slippage. One of the finer points I make in the paper is that variation in the parameter values in the optimization ranges leads to variations in the way ALL the system components interact. Since this system idea comes from someone else, I’m quite sure they set the other parameters based on some form of optimization. Not varying all parameters and sticking with the optimized is a sure way to overly optimistic results.
    3) Its not clear what the theory of market behavior is for this system, probably because it came from someone else. The trading system and associated parameter ranges should be based on some theory of market behavior that will capitalized on. The SPR process helps in the testing of the hypothesis of the theory.

    For more color into some of these comments, Michael Harris posted a dialog on the topic in his blog: http://www.priceactionlab.com/Blog/2016/07/a-few-comments-about-system-parameter-permutation/

    In general SPR is a form of stochastic modeling and much thought and analysis should go into its application. When stochastic modeling is used for insurance applications, there are teams of actuaries that work on the stochastic model. it should not be taken lightly. At its core, SPR is a tool to help system developers avoid a Type-I error (mistakenly beleive a system has value when it doesn’t). Lack of due diligence in setting up the stochastic model not only eliminates any benefit but can enhance false confidence.

    Lastly, this trading system uses only one symbol, the E-mini. My concern with these types of systems when coming from elsewhere is that the symbol choice could have been optimized just like system rules and parameters. I’d run SPR on the system run on similar contracts (e.g. NASDAQ, Russel) before I’d start to draw any performance conclusions.

    • Thanks for the recommendations. I appreciate you taking the time to post your thoughts. I’ll be following up this article with another to accommodate some of your recommendations including to expand the testing of the results to use non-parametric statistics on the final results. In regards to your questions:

      1) I used the entire history of the Emini from 1997 through 2015. I stopped at 2015 to leave the remainder as out of sample. Maybe I misspoke during the video and incorrectly stated 10 years of history but the actual results are based upon the entire history.

      2) I’ll add a stop loss and profit target to the next test. Other than that, this trading model only has two other inputs. This will bring it up to four.

      3) The trading model is taking advantage of the short-term mean reverting nature of the S&P.

      I also like the idea of testing this on other related symbols such as YM, NQ and ER. I also believe most readers will not simply see SRP as an end all to testing. Readers of System Trader Success, most of them anyway, know there are many tests and development process that must be followed before taking a system live. SRP is just one of many tools and techniques but, your concern is noted and appreciated. Thank again!

  • Trevor says:

    Hi Jeff, I want to thank you for making the video which helped me enormously in understanding this approach. Statistics can be a pretty dry subject for an article, or maybe that’s just me, but the video really brought it to life and explained the key points really well.

  • Loser says:

    Hi Jeff,

    One point to add with regard to your video. I have different perspective when calculating how well your system will perform. When you subtract 1 stdev from the mean it really indicates that 84.38% of the time( 68%/ 2 + 50%, where the 50% is the amount to the right of the mean), the system will produce a profit greater than $31,610. For 2 stdev, 97.5% and for 3 stdev – 99.85% of the time the profit will be greater than $24,538.

    What I gather from Walton’s comment is that this test appears to be an indication of parameter sensitivity rather then how well a system will perform.

    Many thanks for the article and video.

    Loser

    • Hey Loser. Thanks for writing. In regards to your first paragraph, I think we’re saying the same thing as I agree with what you wrote. Walton has also recommended to me that using quintiles would be a better approach. In a follow-up article I’m going to look into that. As for your last paragraph I see parameter sensitivity and system performance as the same thing. I see parameter sensitivity having a direct causal relationship to system performance. You’re looking at one side of the coin but I’m looking at the opposite side but, it’s the same coin.

  • Mark says:

    I’m confused. I thought the point of SPP was to get away from anything else that left potential parameter combinations untested. SPR takes random combinations of parameters and tests them, right? The more trials, the closer you get to SPP (all combinations) but there will likely still be some untested.

    Are SPP and SPR to be used together?

    Thanks,
    Mark

    • Mark says:

      I interpret this a bit differently after reading it a second time. Isn’t this just Monte Carlo simulation?

  • geo says:

    Hi Jeff,
    This is a good post and interesting TS code. But wouldn’t running a 1000 iteration optimization in the standard way and then randomly pick 50 out of those iteration, essentially provide the same result?

    Maybe I’m missing something here…

    thanks

    • You’re correct, in this example. But the number of possible outcomes expands dramatically when you increase the number of inputs and input ranges. So, it will depend upon your system. Thus, in some situations you’ll have well over 1,000 possible outcomes. In those cases you can take your 1,000 or 5,000 samples to get an estimate of likely returns.

  • >