Curve-Fitting and Optimization

– Michael Harris, Price Action Lab Blog

The subject of optimization and curve-fitting has received a lot of attention by trading system developers, especially during the last 10 years. There is a lot of confusion about this subject and different views, some even conflicting. There are those who claim that optimization and curve-fitting are unavoidable and necessary but others insist that any trading system that is the outcome of such practices and methods will eventually fail. I have edited a post from last year, in which I present a three-level classification of optimized and curve-fitted systems that I think offers a more systematic look into this important issue.

What is curve-fitting?

In mathematics, curve fitting is the process of finding a curve that fits best a collection of data points in the sense that some objective function subject to constraints is maximized (or minimized). For example, least squares is a curve-fitting method that minimizes the sum of squared residuals. A residual is the difference between a fitted and an actual value. The objective function to minimize when using this method in order to get the best fit is the sum of the squared residuals. Thus, one can immediately notice that what is best is defined only relative to the chosen objective and that curve fitting is essentially the result of optimization.

Next, we will see how the notion of curve fitting applies to trading systems design. A trading system is a process that generates a collection of entry and exit signals. Usually, the algorithm, or model, of a trading system includes a set of parameters. The values of the parameters must be selected so that the system performs best during actual trading. It is common practice to set these parameters by back-testing the system on historical data so that some objective function is maximized (or minimized). For example, one can set the parameters so that the net profit is maximized or the maximum drawdown is minimized, just to mention two possibilities.

Curve-fitting and optimization

When one adopts the definition that trading systems are processes that generate collections of entry and exit signals, then one will realize that what it is done, essentially, when any parameters are adjusted via back-testing is that the timing of the signals is varied so that they are fitted on historical data in such a way so that an objective function is optimized. This is not curve fitting in the usual sense because one is not merely trying to find a curve that best fits the historical data but instead find the best collection of entry signals that in conjunction with the exit signals maximize some objective. This process is much more involved and complicated than plain vanilla curve fitting. It involves selection, or timing of entry and exit signals, that maximize an objective function that is related to performance. It is an optimization problem rather than just a curve fitting problem. As already mentioned, curve fitting may involve optimization but the latter is a process with a much broader scope and includes many more possibilities than curve-fitting. Thus, it is better to refer to optimized systems than to curve-fitted systems although this turns out to be more of a semantics issue for those that understand the process in depth.

For example, let us consider a simple moving average crossover system that generates long entry signals when SMA(t1) > SMA (t2), where t1 and t2 are the periods with t2 > t1, and short entry signals when SMA(t1) < SMA(t2). In its simplest form, this is a stop and reverse system, i.e. when an opposite signal is generated the previous position is closed and reversed. This system cannot be used in practice unless the values of t1 and t2 are selected. One can select those values via optimization of performance using back-testing on historical data. It is a widespread belief that this process results in systems that fail in actual trading because they are “curve fitted”. Is this belief true?

Actually, nobody has ever proven mathematically that the failures of optimized systems, which are well documented, are primarily due to the optimization, or what is commonly referred to as “curve-fitting”. It may be the case that the failures are merely due to the inability of the nature of the algorithms these systems are based on to adapt to changing market conditions. As a matter of fact this is more probable, since most indicators lag price. Thus, it is more probable that optimized trading systems will fail for any values of their parameters at some point. It is the nature of the system and not the optimization that causes the failure. The large class of trading systems based on indicators has high probability of failure but that has been wrongly attributed, based on my experience in developing trading systems, to the optimization process for setting parameters. It does not even matter whether their parameters are set so that small changes in their values result in stable performance. This is not an issue of the integrity of the optimization method used but of the nature of these trading systems. Nevertheless, optimization that causes selection of entry and exit collections is in general a problematic process because it introduces survivorship bias. Selecting collections that performed best in the past overlooks the fact that many such collections failed.

Going back to the simple moving average crossover system, it is easy to understand that given a specific historical data series, changing the values of t1 and t2 will often cause a change in the timing of the entry and exit signals. In this case, selecting any collection of entry and exit signals that results from specific values of the parameters such that some objective function is maximized introduces severe bias because it may be due to chance that the specific collection survived. In the simple moving average crossover example, each collection is completely different from the others in the sense that both the entry and the exit points are different. What can we do to minimize the survivorship bias so that the integrity of the optimization process is not compromised? This question can be answered if we first understand how different types of systems are affected by optimization of their parameters.

A three-level classification of optimized trading systems

We can distinguish three types of systems in relation to how optimization affects their collection of entry and exit points:

Type-I curve-fit: When the parameters of Type-I systems are adjusted both the entry and the exit signals are affected, like in the simple moving average crossover system considered before. In this case, optimization and curve fitting result in collections of entry and exit signals that completely differ from each other and selecting one that performs best introduces maximum bias. These systems have the highest probability of failure.

Type-II curve-fit: When the parameters of Type-II systems are adjusted, only the entry signals are affected. In this case, optimization and curve fitting result in collections of entry and exit signals that differ only in their entry part. Selection introduces less bias than with Type-I systems. These systems have less probability of failure than Type-I systems. Example: Enter long if SMA(t1) > SMA(t2) and Price < P and Exit long at P1 or P2 where P1 and P2 are fixed prices (profit price and stop price).

Type-III curve-fit: When the parameters of Type-III systems are adjusted, only the exit signals are affected. In this case, optimization and curve fitting result in collections of entry and exit signals that differ only in their exit part. Selection introduces less bias than with Type-I or Type-II systems. These systems have the lowest probability of failure because the timing of entry signals is not affected by optimization. Example: Enter long if Close of today > Close of 2 days ago and Exit long at entry price + x points or at entry price – y points, where x and y are the parameter to optimize (profit target and stop-loss).

In general, systems that include indicators involve Type-I curve-fit. Type-II curve fit systems are rarely used in practice. Type-III curve-fit systems include the broad class of systems based on pure price patterns and systems based solely on price action.

Most software programs that discover trading systems automatically generate Type-I systems. It is irrelevant how many statistical tests they perform to measure the significance of the results as these systems have high probability to fail during actual trading because of their nature. An example of a software program that automatically discovers only Type-III systems is Price Action Lab. The search algorithm of this program was designed to search only for price patterns that belong in Type-III class of optimized systems while their selection introduces minimum bias.

As a conclusion we can state that the issue is not whether a system is optimized, because all systems are in one way or another, but to what degree optimization impacts the probability that the system will fail in the future due to its nature. Of course, systems fail for all sorts of other reasons but in this short article we dealt with optimization and curve-fitting. Type-III curve-fit systems, as defined above, appear to have the lowest probability of failure if properly designed.

– Michael Harris, Price Action Lab Blog

About the Author Michael Harris

Michael Harris is a trading expert and a developer of advanced pattern recognition software for the benefit of position and swing traders. Michael developed APS Automatic Pattern Search software which has received great acclaim and recently Price Action Lab, a program that includes an advanced technical analysis indicator based on price patterns, called the p-Indicator. He also provides consulting services about trading system development and market analysis to institutional investors and hedge funds.In years past, Michael has also done work for a number of different financial firms, where he developed a bond portfolio optimization program and trading systems for commodities and stocks. Since 1989, he has been as an active trader.Michael is also a best selling author. His first book “Short-Term Trading with Price Patterns” was published in 1999. His other two books “Stock Trading Techniques with Price Patterns” and “Profitability and Systematic Trading” were published in 2000 and 2008, respectively.

follow me on:
  • Tom says:

    Hi,

    Great article again. I commented on your last article here, hope you remember! You should definitely consider incorporating comments in your price action blog since it hard to find a way to contact you. Does your hotmail address still work? It is listed in one of the books you’ve written.

    Here’s some thoughts on the article. Since I employ a very similar approach in generating trading systems I know the approach is much more robust the plain vanilla genetic algo search but the duration to when the models will work till is unknown. Will it stop working till the next week? Month? The straight forward approach I’ve adopted is to dynamically refresh my models every quarter so that the collection of models will always maintain a certain expectation from a performance perspective. What are your thoughts on this approach?

    Another question that I have about curve fitting: can curve fitting be partially alleviated if we employ a large number of lowly correlated curve fitted models? Say I have 100 models curve fitted to the entire testing history; I don’t do any in sample or out of sample test. I know that each of those models are very lowly correlated but they do from time to time overlap in their signal generation. Now if we traded any single one alone, the probability of the single model not working out of sample is high but if we traded all of them, the probability should be low that the aggregate performance will deviate significantly from in sample. Does this logic make sense? Diversification to curve fitted models.

    Thanks,
    Tom

    • Thank you Tom. There is a contact page for the blog and another one for the main Price Action Lab website.

      You are correct that the time of failure of systems is not known. This is mainly due to new market conditions that impact the expectation negatively. I rebuild models that are based on daily data every year. Models based on intraday one should rebuild more often. Hourly data every 2 months and 5-minute data every month in my opinion.

      Genetic algos do not work because by their nature they curve-fit data to a metric using some indicators and they do that for millions, billions or even trillions of combinations until they find something that satisfies the build objectives. Data-mining bias is so large, that none of the generated systems is significant by any measure. Out-of-sample tests are useless in most cases because if one repeats the process many times, then something will be found at the end that looks good by chance alone.

      My approach minimizes data-mining bias by focusing only on parameter-less price patterns and maximizing a few key parameters, like win rate number of trades: Occam’s razor plus reproducible results. If I run Price Action Lab many times with the same data the same results will be generated with no exception. This is important.

      In my software I have several tools to check significance of systems, such as for example portfolio and robustness tests. Out-of-sample tests are useless in my opinion because their primary objective is for validating the algorithm generating the systems and not the system themselves. This is also important to understand. Instead, I use tests on comparable securities:

      http://www.priceactionlab.com/Blog/2013/05/fooled-by-out-of-sample-testing/

      These tests are more conservative but also more appropriate for trading systems in my opinion. If a system designed for DIA works also in TLT, with these two being mostly anti-correlated instruments, then this means that the patterns have some predictive ability. I am not saying that it is required in all cases. What I am saying is that for me it is another confirmation.

      As far as using many uncorrelated models, it is an interesting idea but as I say in my article the problem is not so much the optimization and curve-fitting but the inability of most indicators to follow changes in market conditions, no matter how the parameters are set. Actually, Parrondo’s paradox says that you can even have two systems that do not work and get from them positive results:

      http://www.priceactionlab.com/Blog/2012/01/i-need-two-losing-investment-strategies-to-win-the-paradox/

      Therefore, it more depends on the logic of those systems but the idea your described is logical and makes sense in principle. I have not tested it because I prefer tests on comparable securities instead.

      Best.

  • Emanuel says:

    Michael and Jeff, thank you for another great article!

    I found this quite interesting and true in my opinion:

    “It may be the case that the failures are merely due to the inability of the nature of the algorithms these systems are based on to adapt to changing market conditions.”

    Michael, I purchased your software Price Action Lab about three months ago and I very satisfied with it. It is a great piece of software and not only because it does what it is supposed to do but primarily because it offers a new perspective and methodology on both systematic and discretionary trading.

    • Thank you Emanuel. The inability of most indicators to properly adapt to changing market conditions is what motivated me to develop a theory about price pattern formation and use it as the foundation of Price Action Lab. In addition, using a small number of performance metrics that do not change allows minimization of curve-fitting and data-mining bias. There is still the problem of reversion to the mean but that is dealt with portfolio backtests.

  • Weldon Kelly says:

    “Example: Enter long if Close of today > Close of 2 days ago and Exit long at entry price + x points or at entry price – y points, where x and y are the parameter to optimize”

    Why aren’t the following considered parameters?
    direction of trade: long
    bar type of expression 1: Close
    time of expression 1: today
    bar type of expression 2: Close
    time of expression 2: 2 days ago
    operator between expressions: >

    Is it because they are constant and not optimized?

  • >