Using System Parameter Randomization To Estimate Future Returns

August 1, 2016 5:00 am10 commentsViews: 1590

You just spent a ton of time creating a trading system and being very careful not to over-optimize. You then tested it on the out-of-sample data segment and the performance looks good. What’s next? Jump right into the live market? Maybe. But instead, you would like to perform one more test called System Parameter Randomization.

The article, System Parameter Permutation – a better alternative?, provided a unique way to estimate possible future returns of a trading system. This method is called System Parameter Randomization (SPR) but, there was no practical example within the article. I like practical examples and I know you probably do too. So, in this article I’m going to take an example trading model from System Trader Success, and follow the guidelines found in the SPR article and estimate our possible future returns. My hope is with this practical example on how to use SPR you just might have a new and valuable tool for your toolbox.

The System

I’m going to use a trading model inspired by Ben Little in the article, The Simplest System You’ll Ever Find for the S&P E-mini. Within this article Ben creates a simple mean reversion strategy for the S&P E-mini which can be traded on a 5-minute chart. Ben carefully tests various filters until he has something workable. The rules I decided upon for this trading model are below:

  • Long trades only
  • Buy only at the lowest low from yesterday
  • The Close of yesterday must be lower than the Open (yesterday was down day)
  • Yesterday’s range must be greater than 18 points (Yesterday was a large range day)
  • The market must be trading below it’s 140 simple moving average (bearish regime filter)
  • $5,000 stop loss
  • $5,000 profit target
  • Close all trades at end of day session

Please note this trading model may not be ready for the live market! It’s an example system and needs to be fully vetted by you to ensure the system performance metrics meet your requirements.

The Trading Environment

  •  25,000 account size
  • Trades 1 contract per trade
  • Only 1 trade per day
  • $20 deducted round trip for slippage and commissions
  • Backtested dates are from 1997 through July 31, 2015.

Results of Backtest

With this information we generate the following results for the trading model.
SPR Baseline EQ Curve

SPR Baseline EQ Curve

SPR Baseline Results

SPR Baseline Results

Let’s say it’s August 1, 2015 and we are happy with our backtested results. In fact, let’s say the last two years of our results were actually executed over the out-of-sample data and we like what we see. We’re now ready to take our system live. But before we do that, we would like to estimate what our likely future returns might be. How are we going to do that? Enter SPR.

Estimating Future Performance With SPR

We know through experience that our system will not likely perform identically to our backtest. We expect our backtest is overly optimistic. When we selected our input values for our trading model we understand this will generate one possible outcome into the future. Furthermore, the likelihood that those values will work identically into the future is not high. This is why we strive to locate stable regions when we do our optimizations because this tends to make our system more robust to market changes. System traders fall into a trap when they backtest to find the best values. That’s not what you should do. You should pick reasonable values within a stable range.

So how does SPR work to estimate future returns?

In essence we’re going to estimate future performance by changing the input values and then recording the results. We will do this many, many times. The input values will actually be picked randomly over a predetermined range. By plugging in random values for our inputs we generate one possible future result of our system. Do this many times and we get a population of possible future outcomes. Once we have this population of possible future outcomes we can calculate the median net profit. This will be our estimate of future performance. Note, you could use any metric you like such as, CAGR, profit factor, or average profit per trade.

By changing the input values and generating hundreds or thousands of different results we are introducing change to the trading model and seeing how it reacts. I would say, we are stressing the trading model. During this stress test we want to know if changing one input parameter makes a significant or small difference to our bottom line. The more our system is able to “hold-up” or perform well over a wide range of values, the more likely it will produce a population of acceptable future outcomes. This population will skew our final median results in favor of a “winning” or acceptable model. If our trading model is negatively affected by changes in the input values we are likely to see a wider range of outcomes that may skew our results in favor of a “poor” or under performing trading model.

Using EasyLanguage To Perform SPR

Here are the steps I’ll use to execute an SPR test on the example system. I’ll be using EasyLanguage, TradeStation, and Excel to perform this test.

1) Determine your critical input parameters and reasonable ranges
2) For each input value generate a random value within the range
3) Apply this set of input values to the backtest and record the results
4) Repeat this test X number of times
5) Once complete, locate the median value of the metric you desire
6) This median value is the estimated future value
Pick Input Values And Ranges

It’s important to use all the inputs when you’re performing the SPR test. In our example, our trading model is not complete. It’s a work in progress and really only two inputs are being used. The following input values and ranges we be used for our SPR test.

  • Our bull/bear regime filter ( 100-200 )
  • Yesterday’s range must be greater than X points (2-30)

The ranges I selected are “reasonable”. What does “reasonable” mean? Good question. Nothing in the trading world is exact so, I can only recommend to pick values that “make sense” in the context of the trading model. Only you will know what that is. It’s probably better to error on the side of providing too wide of range as apposed to a narrow range.

Update The code With Random Number Generator

We’ll write some EasyLanguage code to generate random values for each of our input values. We’ll put these within the “Once” block so the random numbers will only be generated at the start of the run. We don’t want the values changing in the middle of a backtest!

Code which generates random numbers

Code which generates random numbers

TradeStation’s “Random” number function will generate a number between zero and the value you pick. For example, if you use “Random(28)” this will generate a range of integers between 0 and 28. Well, in our case for the BB variable we don’t want a value of zero or one. We want a range between 2 and 30. To do this, we simply have TradeStation generate values between 0 and 28 and then add two. Thus, we generate a range from 2-30.

While we’re editing the code we can also comment out the “BB” and “TrendfilterPeriod” as input values. Why? These are no longer input values but variables which will be randomly chosen. So, let’s remove them from the input section and place them into the “Variables” section of the code. In their place, we’ll need to add a new input value called “Runs”. This will be an input value which we’ll optimize over using TradeStation’s optimization feature. Here is what the inputs look like now:

SPR Inputs

SPR Inputs

Update Code to Store System Results

OK, now we need a way to store the results of a run to a file. This can easily be done by adding the following code to the bottom of our strategy.

SPR File Append Code

SPR File Append Code

Execute!

We now use TradeStation’s optimization feature to iterate over the strategy 1,000 times. Each time the system evaluates the trading model over the backtest, it will be using random values for the parameters we defined. Once the run has completed over the backtest, the results are written to a file and the next iteration starts with a new set of random input values.

Below is an image of the output of our runs. You can see that each row represents a particular run. Each column represents:

  1. The run number
  2. The net profit generated during the test
  3. Our bull/bear regime filter look-back period (trendFilterPeriod)
  4. Points required to be considered a large range day (BB)
SPR Results File

SPR results file

I did not need to include the randomly picked values used during each test. However, I thought it would be interesting to see and would also provide a way to verify the random numbers being picked are correct.

Reviewing The Results: Our Median Return

The results can now be loaded into Excel and the median value calculated. In our case, we determine that our median value is $35,058. This value is positive. Great! But we can go further.

Let’s find the standard deviation and compute a two standard deviation band around our average value. Using Excel we find that the standard deviation is $3,538. Thus, two standard deviations is $7,076.

Our upper band (+2 standard deviations form median) is $42,134 while our lower band (-2 standard deviations from median) is $27,982. We can use the lower band as an estimate on how our system might do into the future assuming poor performance. I’ll call this our pessimistic estimate of future profit. In this case, even our poor performance estimate produces a positive result of $27,982 in net profit.

The key takeaways are:
  • Our median profit is positive
  • Our pessimistic profit is also positive

Based on these numbers it appears our system passes. We randomly changed the input parameters many times over and we see our median net profit over all those different strategy variations remains positive. In fact, even two standard deviations away from the mean remains positive.

Conclusion

So there you have it – an example on how to use SPR to estimate future performance of your system. You can use this technique to estimate potential drawdown, annual returns, or whatever metric you wish. With a little coding and working the numbers in Excel, you just might have another tool for your toolbox to test your system before putting your money on the line.

Video Demo of Using SPR

Downloads

The downloads include the example trading model used in this article with the modified changes to perform the SPR test. Also included is the spreadsheet used to calculate the statistical values. Feel free to download and modify as you wish.

Simple MR Strategy With SPR ( TradeStation ELD, text file )
Simple MR Strategy With SPR ( TradeStation WorkSpace )
Results (Excel XLSX)

Jeff is the founder of System Trader Success – an inBox magazine dedicated to sharing great ideas and concepts from the world of automated trading systems. Read More Google

Facebook Twitter 

Tags:

10 Comments

  • Hi Jeff, thanks for posting this and especially the video. It is very instructive on the mechanics of SPR in TradeStation. However I do want to caution readers that SPR is a tool and as such can be misused resulting in unrealistic expectations. A few caveats:

    1) Only 10 years of price data history was used. This system uses the E-mini so all available price history should be used. I’m not sure how far back TradeStation data goes but I’m sure it is more than 10 years.
    2) Only two parameters are varied in this example. All parameters should be randomized in SPR, not just some subset. The only exception would be the position size parameter. Like I said in the last blog post on this topic: just evaluating a single indicator (or a couple) does not enable enough random interactions. SPR is a method intended for application to a complete system in a portfolio context, meaning inclusion of filters, setups, entries, exits, signal ranking rules, and money management rules and also including commissions and slippage. One of the finer points I make in the paper is that variation in the parameter values in the optimization ranges leads to variations in the way ALL the system components interact. Since this system idea comes from someone else, I’m quite sure they set the other parameters based on some form of optimization. Not varying all parameters and sticking with the optimized is a sure way to overly optimistic results.
    3) Its not clear what the theory of market behavior is for this system, probably because it came from someone else. The trading system and associated parameter ranges should be based on some theory of market behavior that will capitalized on. The SPR process helps in the testing of the hypothesis of the theory.

    For more color into some of these comments, Michael Harris posted a dialog on the topic in his blog: http://www.priceactionlab.com/Blog/2016/07/a-few-comments-about-system-parameter-permutation/

    In general SPR is a form of stochastic modeling and much thought and analysis should go into its application. When stochastic modeling is used for insurance applications, there are teams of actuaries that work on the stochastic model. it should not be taken lightly. At its core, SPR is a tool to help system developers avoid a Type-I error (mistakenly beleive a system has value when it doesn’t). Lack of due diligence in setting up the stochastic model not only eliminates any benefit but can enhance false confidence.

    Lastly, this trading system uses only one symbol, the E-mini. My concern with these types of systems when coming from elsewhere is that the symbol choice could have been optimized just like system rules and parameters. I’d run SPR on the system run on similar contracts (e.g. NASDAQ, Russel) before I’d start to draw any performance conclusions.

    • Thanks for the recommendations. I appreciate you taking the time to post your thoughts. I’ll be following up this article with another to accommodate some of your recommendations including to expand the testing of the results to use non-parametric statistics on the final results. In regards to your questions:

      1) I used the entire history of the Emini from 1997 through 2015. I stopped at 2015 to leave the remainder as out of sample. Maybe I misspoke during the video and incorrectly stated 10 years of history but the actual results are based upon the entire history.

      2) I’ll add a stop loss and profit target to the next test. Other than that, this trading model only has two other inputs. This will bring it up to four.

      3) The trading model is taking advantage of the short-term mean reverting nature of the S&P.

      I also like the idea of testing this on other related symbols such as YM, NQ and ER. I also believe most readers will not simply see SRP as an end all to testing. Readers of System Trader Success, most of them anyway, know there are many tests and development process that must be followed before taking a system live. SRP is just one of many tools and techniques but, your concern is noted and appreciated. Thank again!

  • Hi Jeff, I want to thank you for making the video which helped me enormously in understanding this approach. Statistics can be a pretty dry subject for an article, or maybe that’s just me, but the video really brought it to life and explained the key points really well.

  • Hi Jeff,

    One point to add with regard to your video. I have different perspective when calculating how well your system will perform. When you subtract 1 stdev from the mean it really indicates that 84.38% of the time( 68%/ 2 + 50%, where the 50% is the amount to the right of the mean), the system will produce a profit greater than $31,610. For 2 stdev, 97.5% and for 3 stdev – 99.85% of the time the profit will be greater than $24,538.

    What I gather from Walton’s comment is that this test appears to be an indication of parameter sensitivity rather then how well a system will perform.

    Many thanks for the article and video.

    Loser

    • Hey Loser. Thanks for writing. In regards to your first paragraph, I think we’re saying the same thing as I agree with what you wrote. Walton has also recommended to me that using quintiles would be a better approach. In a follow-up article I’m going to look into that. As for your last paragraph I see parameter sensitivity and system performance as the same thing. I see parameter sensitivity having a direct causal relationship to system performance. You’re looking at one side of the coin but I’m looking at the opposite side but, it’s the same coin.

  • I’m confused. I thought the point of SPP was to get away from anything else that left potential parameter combinations untested. SPR takes random combinations of parameters and tests them, right? The more trials, the closer you get to SPP (all combinations) but there will likely still be some untested.

    Are SPP and SPR to be used together?

    Thanks,
    Mark

  • Hi Jeff,
    This is a good post and interesting TS code. But wouldn’t running a 1000 iteration optimization in the standard way and then randomly pick 50 out of those iteration, essentially provide the same result?

    Maybe I’m missing something here…

    thanks

    • You’re correct, in this example. But the number of possible outcomes expands dramatically when you increase the number of inputs and input ranges. So, it will depend upon your system. Thus, in some situations you’ll have well over 1,000 possible outcomes. In those cases you can take your 1,000 or 5,000 samples to get an estimate of likely returns.

Leave a Reply


Time limit is exhausted. Please reload CAPTCHA.

LEARN HOW TO OPTIMIZE WITHOUT 
CURVE FITTING

Download these four simple steps to help ensure you make money on the live market!

x