One of the biggest issues with system development is that many trading strategies do not hold up into the future. This could be due to several reasons:

- The system is not based on a valid premise
- Market conditions have changed in a dramatic way that invalids the theoretical premises on which the system was developed
- The system has not been developed and tested with a sound methodology. For instance, (a) lack of robustness in a system due to improper parameters, and (b) inconsistent rules and improper testing of the system using out-of-sample and in-sample data

There are several approaches and methodologies to asses robustness and increase likelihood of positive real life trading performance including:

- Peak avoidance
- Limited number of optimizable parameters
- 3D surface charting
- Monte Carlo analysis
- Data scrambling
- Walk forward & walk backward analysis

One of the most robust ways of testing the reliability of a trading system and making sure the program will have the highest likelihood of performing well in real live trading is to use walk forward optimization (WFO), a method first described in the book “Design, Testing, and Optimization of Trading Systems” by Roberto Pardo.

Walk forward analysis is the process of optimizing a trading system using a limited set of parameters, and then testing the best optimized parameter set on out-of-sample data. This process is similar to how a trader would use an automated trading system in real live trading. The in-sample time window is shifted forward by the period covered by the out-of-sample test, and the process is repeated. At the end of the test, all of the recorded results are used to assess the trading strategy.

In other words, walk forward analysis does optimization on a training set; tests on a period after the set and then rolls it all forward and repeats the process. We have multiple out-of-sample periods and look at these results combined. Walk forward testing is a specific application of a technique known as Cross-validation. It means taking a segment of data to optimize a system, and another segment of data to validate. This gives a larger out-of-sample period and allows the system developer to see how stable the system is over time.

The picture below illustrates the walk forward analysis procedure. An optimization is performed over a longer period (the in-sample data), and then the optimized parameter set is tested over a subsequent shorter period (the out-of-sample data). The optimization and testing periods are shifted forward, and the process is repeated until a suitable sample size is achieved.

In order to demonstrate the concept we will perform in this article a walk forward optimization on a volatility breakout trading system (VBO). For the test we will use the German DAX futures, NinjaTrader, CQG historical 1-minute data, and we will assume 3 points of slippage for each R/T trade to cover trading frictions.

The process consists of three main steps:

- Define in-sample and out-of-sample periods
- Define a robust parameters area
- Execute the walk forward

**Definition of the in-sample and out-of-sample periods**

We will choose as in-sample 1/1/2001 to 12/31/2009 for system design and in-sample optimization and 1/1/2010 to 12/31/2012 as out-of-sample period to evaluate the in-sample optimization robustness and execute the walk forward. We will then use a 3:1 ratio for the WFO (walk forward optimization):

- Optimize 2007 to 2009 and verify performance out-of-sample in 2010
- Optimize 2008 to 2010 and verify performance out-of-sample in 2011
- Optimize 2009 to 2011 and verify performance out-of-sample in 2012

**Define the robust parameters area in the in-sample period**

In this section we will define the “robust area” of the system parameters. We will optimize only 3 system parameters:

- Lookback period of the fast average
- Lookback period of the slow average
- Volatility filter

** Other system parameters that we will not optimize are:**

- Start time: 09:00 (GMT+1)
- End time: 22:00 (GMT+1)
- Last trade: 18:00 (GMT+1)
- Stop risk: 2%
- Maximum trades per day: 3
- Exit on close: True

As we can see the average SQN (system quality number) tends to decrease as we increase the time period of the chart in minutes. We will choose **14 minutes **for all simulations going forward.

For more info on SQN:

http://www.traderivar.com/2010/03/sqn-trading-system-quality-number.html

The NinjaTrader code for the system quality number can be downloaded here:

http://www.vbosystems.info/download.html

Via a 3D surface chart we can identify the robust parameter area as follows:

- Slow average: 12 to 30 minutes
- Fast average: 330 to 500 minutes

We define “robust” a parameter surface area that does not have major peaks or valleys and has generally a good performance.

**Definition of the robust parameter for the volatility filter**

As we can see the robust area for the filter is between 0.55 and 0.70 when the SQN is slowly increasing.

Now that we have identified the robust parameters area it is worth performing a full in-sample optimization to see how the system would have performed between 2001 and 2009.

The system generated a net profit of $120,000 between 2001 and 2009 with a profit factor of 1.56, performing 756 trades with an average of 41% trades profitable. The systems exhibits certain desirable characteristics such as a high win/loss ratio of 2.28.

As anticipated we will now proceed with a walk forward optimization.

**Step 1**: We’ll optimize between 2007-2009 and find the best parameters.

The best parameters for the 2007-09 period are:

- Fast average: 12
- Slow average: 410
- Filter: 0.55

We apply these parameters to the out-of-sample period in 2010 with the following results:

Net profit: $12,300

DD: $9,000

% profitable: 43%

Profit factor: 1.38

**Step 2**: We’ll optimize between 2008-2010 and find the best parameters.

The best parameters for the 2008-10 period are:

- Fast average: 20
- Slow average: 500
- Filter: 0.70

We apply these parameters to the out-of-sample period in 2011 with the following results:

Net profit: $27,900

DD: $7,450

% profitable: 43%

Profit factor: 1.61

**Step 3**: We’ll optimize between 2009-2011 and find the best parameters.

The best parameters for the 2009-11 period are:

- Fast average: 20
- Slow average: 420
- Filter: 0.55

We apply these parameters to the out-of-sample period in 2012 with the following results:

Net profit: $17,540

DD: $7,300

% profitable: 41%

Profit factor: 1.58

In this article we have shown how to perform a walk forward optimization on an intraday mechanical system. The walk forward results are in line with the in-sample results and this builds confidence in the robustness of the strategy.

For more information on algorithmic trading strategies please visit: http://www.vbosystems.info

— By Amon Licini of VbO Systems. **VbO Systems** is a developer of 100% automated trading systems coded in NinjaTrader that can be auto-traded on almost all asset classes. Amon Licini, the founder of Vbo Systems, has been a private trader for 15 years and a senior manager with various corporates in Italy. Amon’s main trading interests lie in the area of volatility and open range breakouts for intraday systems. He lives in Milan with his wife and 2 children and loves traveling when he’s not developing new systems. Amon has a degree in mechanical engineering from the Polytechnic University of Milan.

Contributing authors are active participants in the financial markets and fully engrossed in technical or quantitative analysis. They desire to share their stories, insights and discovers on System Trader Success and hope to make you a better system trader. Contact us if you would like to be a contributing author and share your message with the world.

**Session expired**

Please log in again. The login page will open in a new tab. After logging in you can close it and return to this page.

At each step of the WFA, the last out-of-sample becomes part of the in-sample and this enforces hindsight bias, meaning that only the last test is a pure out-of-sample one. This renders the WFA useless and it reduces it to the usual in-sample followed by out-of-sample testing.

Regardless of this fundamental mistake, you have only shown an example where 3/3 out-of-sample were good. What do you do when k/n samples are good? How do you judge the performance of WFA?

Lastly, the paper is missing an important comparison with a simple in-sample/out-of-sample test.

I disagree, Bob.

The concatenated equity curve is valid because each portion of it is generated as an OOS test. Yes, the last OOS becomes part of the IS but this does not introduce hindsight bias because it’s then being tested on data that has not been studied before.

I’m not sure that I understand you Bob . . .

Each (simulated) position will be based only on information that would actually have been available at that time – there is no “data snooping” involved.

By “hindsight bias”, all that is happening is that the in-sample data window is constantly expanding. If you assume that more data and a larger sample size is more meaningful, then this is a good thing. “Hindsight Bias” is an odd term – if you don’t expect a market to continue to behave in the future in approximately the same way that it has behaved in the past, then it may be very hard to trade! Moreover, you would have no reason to become embroiled in any kind of optimization process (walk-forward or simple) in the first place.

Hello there, I have a question:

After you have done the walk-forward optimization, which set of parameters will you use for your live trading?

The parameters of the last walk-forward period?

The parameters that gave the best results?

The parameters where the out-of-sample results were the best?

The parameters where the out-of-sample results were closest to the in-sample results?

For WFO I would use the most recent optimization. In the future at the end of current OOS period, you would optimize again and use those newly generated values on the next OOS period.

Hi Eddi,

this is a very good question! As far as I see Pardo himself (at least in “The Evaluation and Optimization of Trading Strategies”) doesn’t touch the issue explicitly. However, he stresses the point that the results of a WFA only have a limited shelf life and need regular readjustment (based on another optimazation). That’s why I decided to use the results of the last walk-forward-window.

Howard Bandy by the way also suggests taking the “best current (!) model” (see: https://www.youtube.com/watch?v=oN10-9oIsDY; minute: 34.30).

Best regards Oliver