# Robustness of Algorithmic Trading Systems (That Work)

This is the first of a series of articles that will discuss in depth the topic of algorithmic trading systems for retail investors with particular attention to: optimization and curve fitting, market selection, backtesting, walk-forward testing, portfolio creation, intraday algorithmic trading, and so forth.

In this first article we will discuss how to verify if a certain trading system is robust or not.

As defined in Wikipedia “[…] robustness defines the ability of a financial trading system to remain effective under different markets and different market conditions, or the ability of an economic model to remain valid under different assumptions, parameters and initial conditions.”

*Source: **en.wikipedia.org/wiki/Robustness_(economics)*

There are four main tests to assess the robustness of an intraday trading system:

- Does the system work on a variety of parameter combinations?
- Does the system work on a variety of time frames?
- Does the system work on a variety of instruments?
- Does the system work on at least 6 years of past data without the need to be re-optimized frequently?

Let’s take a simple volatility breakout intraday system to make an example on ES (S&P 500 Mini). This is a simple system with only two key parameters that can be optimized: PeriodS (the lookback period in bars of a slow indicator) and PeriodF (the lookback period in bars of a fast indicator). The system has the following characteristics:

- Intraday
- Entry: fast move with increase in volatility
- Exit: stop loss, change of trend, end-of-day
- Risk management: volatility based stop loss, trailing stop loss, range contraction filter, limitation on max number of trades per day
- Position sizing: 1 contract

**Does the system work on a variety of parameter combinations?**

In order to check robustness of the variety of parameter combinations we optimize PeriodS from 100 bars to 500 bars with steps of 25 and PeriodF from 10 bars to 40 bars with steps of 10.

We can see that the SQN average is 4.4, the minimum 1.9, and the maximum 5.8.

The System Quality Number (SQN) is an indicator of a system quality based on the statistical t-score popularized by Van Tharpe and defined as:

*SQN= Squareroot (N) Average (of the N Profit&Loss) / Std Dev (of the N Profit&Loss).*

Basically it is the average trade, divided by the standard deviation of the average trade and multiplied by the square root of the number of trades. Hence: many trades, high average trade, and low volatility of the average trade will provide a higher SQN.

You can find more information below. Values over 2.5 normally indicate a good system:

http://www.traderivar.com/2010/03/sqn-trading-system-quality-number.html

The code for NinjaTrader is available freely for download at our site: www.vbosystems.info/download.html

The analysis above shows this system is robust across a variety of parameter combinations.

**Does the system work on a variety of time frames?**

The second test relates to time frames. A robust system will maintain good results across a variety of time frames. In order to check robustness over various time frames we keep the two parameters fixed at 300 (PeriodS) and 30 (PeriodF) and optimize the time frame from 10 to 50 bars with steps of 5.

We can see that the lowest SQN value is 3.6 with a time frame of 25. The charts indicates that this system is most effective at lower and higher time frames, however it maintains a good SQN value throughout.

**Does the system work on a variety of instruments?**

A robust system will maintain good results across a variety of instruments. The most robust systems will have good results with the exact same parameters across many instruments. At the very least one should focus on systems that can actually work across different instruments, even though the best parameters may be different.

In order to check robustness over different instruments we optimize on 27-minute bars PeriodS from 100 bars to 600 bars with steps of 25 and PeriodF from 5 bars to 50 bars with steps of 5 across a basket of 12 futures contracts diversified across indexes, forex, energy, and interest rates: 6B, 6E, CL, EMD, ES, FDAX, FGBL, IBEX35, NG, RT, TF, ZB. The optimization provides the following results:

The results above, from January 2006 to April 2013, show this particular system can be profitable – with different parameters – on several futures markets. However, the most robust systems will maintain good results across different tradables whilst maintaining the same parameters. If we analyze the heatmap of the average SQN across all tradables, we will see the following results:

The table above shows that the average SQN across all tradables, across all parameters, is relatively stable, with a minimum value of 2. In order to find the most stable parameters the best option is to divide the average SQN by its standard deviation like in the following heatmap:

The most stable parameter combination across all 12 futures is PeriodF=15 and PeriodS=425 which provides the following results:

The results above indicate this system has robust results across 12 futures contracts diversified across indexes, forex, energy, and interest rates.

**Does the system work on at least 6 years of past data without the need to be re-optimized frequently?**

A robust system will maintain good results over various years without the need to be re-optimized frequently. Let’s consider the parameters from the first test that provided the best SQN: PeriodS=150 and PeriodF=30. The results on ES 15-minute bars are as follows:

The number of trades is pretty consistent every year (except 2013 as the data is only until mid April) as well as the %win.

In conclusion, when evaluating the robustness of a trading system there are four key tests to perform. Only if a system performs well across a variety of parameter combinations, a variety of time frames, a variety of instruments and over at least 6 years of past data without the need to be re-optimized frequently it can be considered fully robust.

All optimizations described in this article have been done using NinjaTrader and Kinetick 1-minute data. For more information about trading systems please visit our website: www.vbosystems.info.

— By Amon Licini of VbO Systems. **VbO Systems** is a developer of 100% automated trading systems coded in NinjaTrader that can be auto-traded on almost all asset classes. Amon Licini, the founder of Vbo Systems, has been a private trader for 15 years and a senior manager with various corporates in Italy. Amon’s main trading interests lie in the area of volatility and open range breakouts for intraday systems. He lives in Milan with his wife and 2 children and loves traveling when he’s not developing new systems. Amon has a degree in mechanical engineering from the Polytechnic University of Milan.

2:15 pm

I disagree with a couple things said here.

First, if a system works on one and only one ticker then this post suggests it is not robust and should not be traded. I can envision a ticker whose trading is dominated by a handful of well-capitalized institutions. Suppose most of those institutional traders use a particular indicator. A system using that indicator is likely to be profitable–and will continue to be profitable as long as those institutional traders continue trading that way. Regardless of whether other institutional traders directing the trading of other tickers use that indicator or not (i.e. the system is profitable on other tickers), it will continue to work.

My second disagreement is about “frequent re-optimization.” I do agree that less frequent optimization is preferred, simpler, seemingly less curve-fit, etc. However, isn’t is quite conceivable to have a profitable system using a walk-forward routine that uses two years IS and six months (choose any “frequent” time interval) OOS?

I have a bigger problem with WFO in general because some combinations of IS/OOS time intervals may work whereas others may not. If you choose the ones that work and trade live then are you curve-fitting there?

4:36 pm

Very interesting article, do you read about monte carlos simulations and van tharp theorie? maybe this could be interesting too for you. really good, thanks.