Learn To STOP Curve Fitting!
Download this free guide on how to stop curve fitting. Following these four simple steps can improve your trading dramatically!
You just spent a ton of time creating a trading system and being very careful not to over-optimize. You then tested it on the out-of-sample data segment and the performance looks good. What’s next? Jump right into the live market? Maybe. But instead, you would like to perform one more test called System Parameter Randomization.
I’m going to use a trading model inspired by Ben Little in the article, The Simplest System You’ll Ever Find for the S&P E-mini. Within this article Ben creates a simple mean reversion strategy for the S&P E-mini which can be traded on a 5-minute chart. Ben carefully tests various filters until he has something workable. The rules I decided upon for this trading model are below:
Please note this trading model may not be ready for the live market! It’s an example system and needs to be fully vetted by you to ensure the system performance metrics meet your requirements.
We know through experience that our system will not likely perform identically to our backtest. We expect our backtest is overly optimistic. When we selected our input values for our trading model we understand this will generate one possible outcome into the future. Furthermore, the likelihood that those values will work identically into the future is not high. This is why we strive to locate stable regions when we do our optimizations because this tends to make our system more robust to market changes. System traders fall into a trap when they backtest to find the best values. That’s not what you should do. You should pick reasonable values within a stable range.
So how does SPR work to estimate future returns?
In essence we’re going to estimate future performance by changing the input values and then recording the results. We will do this many, many times. The input values will actually be picked randomly over a predetermined range. By plugging in random values for our inputs we generate one possible future result of our system. Do this many times and we get a population of possible future outcomes. Once we have this population of possible future outcomes we can calculate the median net profit. This will be our estimate of future performance. Note, you could use any metric you like such as, CAGR, profit factor, or average profit per trade.
Here are the steps I’ll use to execute an SPR test on the example system. I’ll be using EasyLanguage, TradeStation, and Excel to perform this test.
It’s important to use all the inputs when you’re performing the SPR test. In our example, our trading model is not complete. It’s a work in progress and really only two inputs are being used. The following input values and ranges we be used for our SPR test.
The ranges I selected are “reasonable”. What does “reasonable” mean? Good question. Nothing in the trading world is exact so, I can only recommend to pick values that “make sense” in the context of the trading model. Only you will know what that is. It’s probably better to error on the side of providing too wide of range as apposed to a narrow range.
We’ll write some EasyLanguage code to generate random values for each of our input values. We’ll put these within the “Once” block so the random numbers will only be generated at the start of the run. We don’t want the values changing in the middle of a backtest!
TradeStation’s “Random” number function will generate a number between zero and the value you pick. For example, if you use “Random(28)” this will generate a range of integers between 0 and 28. Well, in our case for the BB variable we don’t want a value of zero or one. We want a range between 2 and 30. To do this, we simply have TradeStation generate values between 0 and 28 and then add two. Thus, we generate a range from 2-30.
While we’re editing the code we can also comment out the “BB” and “TrendfilterPeriod” as input values. Why? These are no longer input values but variables which will be randomly chosen. So, let’s remove them from the input section and place them into the “Variables” section of the code. In their place, we’ll need to add a new input value called “Runs”. This will be an input value which we’ll optimize over using TradeStation’s optimization feature. Here is what the inputs look like now:
OK, now we need a way to store the results of a run to a file. This can easily be done by adding the following code to the bottom of our strategy.
We now use TradeStation’s optimization feature to iterate over the strategy 1,000 times. Each time the system evaluates the trading model over the backtest, it will be using random values for the parameters we defined. Once the run has completed over the backtest, the results are written to a file and the next iteration starts with a new set of random input values.
Below is an image of the output of our runs. You can see that each row represents a particular run. Each column represents:
I did not need to include the randomly picked values used during each test. However, I thought it would be interesting to see and would also provide a way to verify the random numbers being picked are correct.
The results can now be loaded into Excel and the median value calculated. In our case, we determine that our median value is $35,058. This value is positive. Great! But we can go further.
Let’s find the standard deviation and compute a two standard deviation band around our average value. Using Excel we find that the standard deviation is $3,538. Thus, two standard deviations is $7,076.
Our upper band (+2 standard deviations form median) is $42,134 while our lower band (-2 standard deviations from median) is $27,982. We can use the lower band as an estimate on how our system might do into the future assuming poor performance. I’ll call this our pessimistic estimate of future profit. In this case, even our poor performance estimate produces a positive result of $27,982 in net profit.
Based on these numbers it appears our system passes. We randomly changed the input parameters many times over and we see our median net profit over all those different strategy variations remains positive. In fact, even two standard deviations away from the mean remains positive.
So there you have it – an example on how to use SPR to estimate future performance of your system. You can use this technique to estimate potential drawdown, annual returns, or whatever metric you wish. With a little coding and working the numbers in Excel, you just might have another tool for your toolbox to test your system before putting your money on the line.
The downloads include the example trading model used in this article with the modified changes to perform the SPR test. Also included is the spreadsheet used to calculate the statistical values. Feel free to download and modify as you wish.
Jeff is the founder of System Trader Success – a website and mission to empowering the retail trader with the proper knowledge and tools to become a profitable trader the world of quantitative/automated trading.
Please log in again. The login page will open in a new window. After logging in you can close it and return to this page.