## Learn To STOP Curve Fitting!

Download this free guide on how to stop curve fitting. Following these four simple steps can improve your trading dramatically!

Trading financial instruments in an objective systematic fashion has numerous advantages over subjective approaches:

- Intelligently designed automated trading systems can and often do outperform human driven trading due to various cognitive biases and emotionalism.
- An effective data-mining program can discover subtle patterns in market behavior that most humans would not have a chance of seeing.
- An automated system is absolutely repeatable, while a human-driven system is subject to human whims. Consistency of decision-making is vital long-term profitability. Repeatability is also valuable because it allows examination of trades in order to study operation and perhaps improve performance via signal filtering.
- Most properly designed automated trading systems are amenable to rigorous statistical analysis that can assess performance measures such as expected future performance and the probability that the system could have come into existence due to good luck rather than true power.
- Unattended operation is possible.

Automated trading systems are usually used for one or both of two applications. *TSSB (Trading System Synthesis and Boosting) *is a state-of-the-art program that is able to generate both applications: (1) a complete, stand-alone trading system which makes all trading decisions and (2) a model which may be used to filter the trades of an existing trading system in order to improve performance. We refer to this as “boosting”. It is often the case that by intelligently selecting a subset of the signals generated by an existing trading system, and rejecting the others, we can improve the risk/reward ratio.

Whether the user’s goal is development of a stand-alone trading system or a filtering system to boost the performance of an existing trading system, there are two common approaches to its development and implementation: (1) *rules-based *(IF/THEN rules proposed by a human) and *predictive modeling*.

A rules-based trading system requires that the user specify the exact rules that make trade decisions, although one or more parameters associated with these rules may be optimized by the development software. Here is a simple example of an algorithm-based trading system:

**IF **the short-term moving average of prices exceeds the long-term moving average of prices, **THEN **hold a long position during the next bar.

The above algorithm explicitly states the rule that decides positions bar-by-bar, although the exact definition of ‘short-term’ and ‘long-term’ is left open. The developer might use software to find moving-average lookback distances that maximize some measure of performance. Programs such as TradeStation® include a proprietary language (EasyLanguage® in this case) by which the developer can specify trading rules.

With the widespread availability of high-speed desktop computers, an alternative approach to trading system development has become feasible. *Predictive modeling *employs mathematically sophisticated software to examine indicators derived from historical data such as price, volume, and open interest, with the goal of discovering repeatable patterns that have predictive power. A predictive model is essentially a mathematical or logical formulate that relates these patterns to a forward-looking variable called a *target *or dependent variable, such as the market’s return over the next week. This is the approach used by *TSSB, *and it has several advantages over algorithm-based system development:

- Intelligent modeling software utilizing machine learning can discover patterns that are so complex or buried under random noise that no human could ever see them.
- Once a predictive model trading system is developed, it is usually easy to tweak its operation to adjust the risk/reward ratio to suit applications ranging across a wide spectrum. It can obtain a desired trade off between numerous signals with a lower probability of success and fewer signals with a higher probability of success. This is accomplished by adjusting a threshold that converts model predictions into discrete buy and sell signals.
- Well designed software allows the developer to adjust the degree of automation employed in the discovery of trading systems. Experienced developers can maintain great control over the process and put their knowledge to work creating systems having certain desired properties, while inexperienced developers can take advantage of massive automation, letting the software have majority control.
- In general, predictive modeling is more amenable to advanced statistical analysis than rules-based system development. Sophisticated analysis algorithms to test the statistical soundness of its discoveries can be incorporated into the model-generating process more easily than they can be incorporated into systems based on human-specified rules.
- Predictive modeling a well developed mathematical discipline for extracting maximum information from a data set that compliment human intuition. Intuition is able to propose data series and ways of transforming them into a large list of candidate indicators. Predictive modeling, even its simplest form, linear regression is superior to human intuition in selecting the best candidates and combining them into a prediction. There have been over academics 150 studies comparing human experts to statistical models attesting to this fact.

The predictive modeling approach to trading system development relies on a basic property of market price movement: all markets contain patterns that tend to repeat throughout history, and hence can often be used to predict future activity. For example, under some conditions a trend can be expected to continue until the move is exhausted. Under other conditions, a different pattern manifests: a trend is more likely to be followed by a retracement toward the recent mean price. A predictive model studies historical market data and attempts to discover the features that discriminate these two patterns.

The goal of predictive modeling then is finding patterns that repeat often enough to be profitable. Once discovered, the model will be on the lookout for the pattern to reoccur. Based on historical observations, the model will then be able to predict whether the market will soon rise, fall, or remain about the same. These predictions can be translated into buy/sell decisions by applying thresholds to the model’s predictions.

Predictive models do not normally work with raw market data. Rather, the market prices and other series, such as volume, are usually transformed into two classes of variables called *indicators* and *targets. *This is the data used by the model during its training, testing, and ultimate real time use. It is in the definition of these variables that the developer exerts his or her own influence on the trading system.

Indicators are variables that look strictly backwards in time. When trading in real time, as of any given bar an indicator will be computable, assuming that we are in possession of sufficient historical price data to satisfy the definition of the indicator. For example, someone may define an indicator called *trend *as the percent change of market price from the close of a bar five bars ago to the close of this bar. As long as we know these two prices, we can compute this *trend* indicator. *TSSB *can compute over a hundred types of indicators that quantify numerous features of market behavior.

*Targets *are variables that look strictly forward in time. (In classical regression modeling, the target is often referred to as the *dependent variable.) *Targets reveal the future behavior of the market. We can compute targets for historical data as long as we have a sufficient number of future bars to satisfy the definition of the target. Obviously, though, when we are actually trading the system we cannot know the targets unless we have a phenomenal crystal ball. For example, we may define an indicator called *day_return *as the percent market change from the open of the next day to the open of the day after the next. If we have a historical record of prices, we can compute this target for every bar except the last two in the dataset. *TSSB *can compute a variety of target variable types.

In summary, the fundamental idea behind predictive modeling is that *indicators *may contain information that can be used to predict *targets. *The task of predictive model is to find and exploit any such information.

— By David Aronson

Part 2 of this series can be found here, Predictive-Model Based Trading Systems Part II

David Aronson is a pioneer in machine learning and nonlinear trading system development and signal boosting/filtering. Aronson is Co-designer of TSSB (Trading System Synthesis and Boosting) a software platform for the automated development of statistically sound predictive model based trading systems. He has worked in this field since 1979 and has been a Chartered Market Technician certified by The Market Technicians Association since 1992. He was an adjunct professor of finance, and regularly taught to MBA and financial engineering students a graduate-level course in technical analysis, data mining and predictive analytics. His recently released book, Statistically Sound Machine Learning for Algorithmic Trading of Financial Instruments, is a in-depth look at developing predictive-model-based trading systems using TSSB.

Contributing authors are active participants in the financial markets and fully engrossed in technical or quantitative analysis. They desire to share their stories, insights and discovers on System Trader Success and hope to make you a better system trader. Contact us if you would like to be a contributing author and share your message with the world.

**Session expired**

Please log in again. The login page will open in a new window. After logging in you can close it and return to this page.

Perhaps I’m not the target consumer, but I don’t have the time to read the book or the manual. How does this software compare to other free/cheap options like TradeStation? I was not able to find a comparison table or a clear statement of the advantages.

Hello Stan. It is very different than TradeStation. In fact, they serve totally different goals. Overall TSSB software is not a trading platform, charting program or a development environment where you, as a programmer, write code. Instead, TSSB is a stand-alone tool for researching statistically sound predictive model based trading systems via machine learning. It’s more like Adaptrade Builder than TradeStation. In short, it’s a stand-alone tool.

The authors rediscovered the wheel. These program are available for a long time. What they described as being predictive modelling is a process called data-snooping which appears to be predictive but it is essentially a statistical bias from using the data many times. Probably this is the reason they offer such program for free and they want to charge high consultation fees because if the program could make money they would not give it away. Overall, this article/promotion was disappointing and the quality of this website has gone down the drain.

Sorry you feel the quality of the site is not to your standards. I for one was happy to expose my readers to the concept of predictive modeling as a way to discover market edges and/or build complete trading models. As for the software, I’m pleased there is such a free research tool. Clearly a lot of work went into it and the user manual is very extensive. I wish I had something like this years ago. I’m not sure how a free tool, which can aid in the development of trading systems, be viewed as “gone down the drain?”. FYI, I make no money from the book or software. Maybe if you have a specific criticism on the software the author could respond directly.

Sorry but the quality of this website started showing deterioration signs after the article on “best stock market indicator” but also before that.

I pointed to a specific criticism of the software which it seems you missed and the author avoided talking about it and it is data-snooping bias.

“What they described as being predictive modelling is a process called data-snooping which appears to be predictive but it is essentially a statistical bias from using the data many times.

I am glad you take criticism seriously. I think your blog was way better when you wrote the articles and you did not have any contributors.

Thanks for writing Bob. I have contributors write material to provide different perspectives as I wish STS to be beyond my personal knowledge and experience.

Bob’s post reveals he has neither looked at TSSB or the book. He does realize that data mining incurs a bias from repeated looks at the data. The fact that no other software of which I am aware deals with this problem was the prime motive for creating TSSB. I contains statistically rigorous methods for generating unbiased results and statistical significance tests that are robust to data mining bias. The latter, the Monte Carlo Permutation test was first proposed by my co-author Dr. Tim Masters for, who holds a doctorate in statistics and numerical computing. Thus the entire point of TSSB is deal with the problem that Bob correctly points out.

The two articles I posted did not mention this problem but it is given extensive treatment in our book and software.

David Aronson

Dear Sir,

Monte Carlo tests cannot deal with data-mining bias because they cannot reject perfectly fitted systems. Neither you nor Masters are aware of that but we do because we actually trade.

“The main limitation of the Monte Carlo analysis are that if your initial results are curve – fitted your results will be a nonsense. The Monte Carlo analysis is good only when it is applied for a sound trading system and not to an over-fitted one.”

http://www.fxhackers.com/2012/11/monte-carlo-simulation-for-trading.html

Also I am surprised you mix data-mining and data-snooping bias. These are two different things. Your program works by actually enforcing data-snooping bias and there is not sound known statistical tests other than the SPA test to deal with that. All that you can show a superior rule exists in the universe of rules you deal with but it is impossible to know which one it is. If you know of a test that can determine whether a specific trading rule will work in the future and it is not just a result of data-mining please let us know so we can contact Stockholm and nominate you for the Nobel Prize.

Joe

Can you tell me what is your understanding of how TSSB makes use of the Monte Carlo method. From your response it appears as if you are thinking of another application of Monte Carlo that we do not employ. I am happy to respond further but let first see if we are on the same page.

I am familiar with Hansen’s SPA test. In fact I discussed it in my first book Evidence Based Technical Analysis (Wiley 2006) pages 329-330)

thanks

David Aronson

Hello again Bob (sorry about Joe) can you tell me what you understand to be the distinction between data mining bias and data snooping bias.

thanks

David Aronson

“Can you tell me what is your understanding of how TSSB makes use of the Monte Carlo method.”

We are familiar with Dr. Masters work and aware that he makes various significant and unfounded assumptions like for example that all random pairing of system returns with daily raw returns are equally likely, something that amounts to esoteric statistics in this particular case. Then his methods apply only to systems that are invested all the time otherwise considering daily raw returns in a MC or bootstrap does not make sense. As someone commented in Amazon about your book, these are statistical acrobatics because there is no sound theoretical framework that underpins it.

“Hello again Bob (sorry about Joe) can you tell me what you understand to be the distinction between data mining bias and data snooping bias.”

This is reversal of the burden. You claim to be able to deal with these so please you help us clarify the distinction. Obviously, you use thew two terms interchangeably which is not correct.

To conclude, if your statistical methods worked it would only make sense to have kept them secret. Otherwise, revealing them to the public and giving the software away for free is not something that someone with an edge would do.

Bob

Actually most people in the field of machine learning and data mining do use the terms interchangeably. Robert Hansen does in his most recent article on SPA (2010). I recently chaired the machine-learning track of a conference for traders in Chicago sponsored by Terrapin and there as well speakers used them interchangeably. While I did make a distinction between data mining and data snooping in my 2006 book, the later relating to obtaining hypotheses from others research papers, I came to find others did not make that distinction. You seem to think it’s crucial. I invite to to explain the difference if you care to if not then don’t.

You’ve made erroneous assumptions about the type of Monte Carlo Permutation tests used in TSSB. You seem to think that we are permuting the order of trades produced by an over fitted trading system. Moreover your assumptions about Dr. Masters current work are also wrong. His earlier paper, which is posted on the Hood River website has nothing to do with the more advanced MCP test used in TSSB. This latest version has not be put in the public domain.

The version of the MCP test discussed in my 2006 book Evidence Based Technical Analysis, Hansen’s SPA test and White Reality Check are all limited in the same way. They do not work for machine-learning, which involves a guided search through the hypothesis space. These tests only work if the search is either exhaustive or random. It is precisely for this reason that the MCP test incorporated into TSSB was designed to be valid under the condition of a guided searche ( machine learning).

Your prior posts suggest your opinions are entrenched and any further attempt to correct your errors will prove fruitless. I will leave you the last word as you seem to need that. I have no doubt it too will be filled with inaccuracies and insults. Have at it.

David Aronson

Below is response from Dr. Timothy Master. He will not be responding further.

Bob is missing the entire point of the statistical tests such as MCPT in TSSB. He is correct in asserting that curve fitting, learning noise instead of authentic patterns, et cetera, result in an optimistic bias in performance figures. But what he does not seem to understand is that the main driving force behind the design of TSSB is to defeat this exact problem! The version of the MCPT documented in the several papers written by me and available online is an early design, excellent in some narrowly defined applications but worthless in a broader environment. (The paper clearly states this, and lays out some ground rules for when it is and is not applicable.) This algorithm has subsequently been refined to be far more robust, and it is this refined version that is incorporated into TSSB.

In a general sense, here is how a properly designed MCPT test operates. Stipulate that the process of developing a model-based trading system (training models and committees, setting thresholds, selecting the best performers, et cetera) produces optimistically biased performance figures. If the models are excessively powerful and overfit the data, this bias can be extreme. So to assess and account for this we deliberately destroy any predictive power in the data. This may be done by randomly permuting market changes, permuting targets relative to indicators, permuting predicted market moves relative to optimized thresholds, or in any other number of ways, each of which has its benefits and limitations. If this is done a large number of times (hundreds or thousands) we can assess the degree to which the system development process inflates performance. If, for example, we see that the performance results obtained from the actual data greatly exceeds that produced from the permuted data, we can be confident that our developed system has true power above and beyond that produced by the bias inherent in the development cycle.

These issues are discussed in greater detail in the the book Statistically Sound Machine Learning. Pages 161-178 describe a particular MCPT useful for quick-and-dirty screening of indicators for predictive power. Page 206 describes another MCPT that can be used to test the true predictive power (that above and beyond any data mining bias) of an individual model. Pages 299-312 describe an extremely powerful MCPT that completely removes the effects of data mining bias throughout the entire chain of decision making (models, committees, oracles) in order to assess the true capabilities of the system. In fact, Pages 302-305 describe how it can even account for the interaction between long-term market bias and trading systems that are unbalanced in their long-short positions. Finally, the section “Data Snooping… Friend or Foe” on Page 444 provides a big-picture discussion of the relevant issues.

I do make the point several times in the book that many of these algorithms are recent developments and lack rigorous theoretical justification. On the other hand, I do have a PhD in mathematical statistics. I’ve authored five books on the subject, all of which have received rave reviews in the professional peer press (and generally very positive reviews in the much less reliable Amazon.com site). And I’ve continuously worked as a professional consultant to the market-trading community since 1995. I will say that I use the algorithms in TSSB in my own work and I have full confidence in their veracity. I cannot guarantee that they are completely correct in every possible circumstance, but my experience has been that they are worthy of trust in every application in which I’ve applied them.

Finally, let me say that if anyone has a specific technical criticism of any of these algorithms, I would be happy to hear this criticism, and if I think it is valid I will answer in some way. Critics may, in fact, help me to improve the program. I am always eager for constructive criticism. On the other hand, I will not respond to vague generalities such as, “He doesn’t know what he’s talking about.” This is especially true if the criticism comes from someone who obviously has not even studied the book or program and hence is just speculating. Discussions of this nature are in no one’s best interest.

Tim

“If, for example, we see that the performance results obtained from the actual data greatly exceeds that produced from the permuted data, we can be confident that our developed system has true power above and beyond that produced by the bias inherent in the development cycle.”

Again, you are both trying to accuse me of plain criticism but the truth is I have tried to explain to you a few times that a perfectly curve fitted system will always appear good based on your tests. So you are basically achieving nothing. This is a fundamental problem with MC simulation and no matter how one modifies it it will be there.

“our prior posts suggest your opinions are entrenched and any further attempt to correct your errors will prove fruitless. ”

I was going to say this about your posts. MC simulation can rule out random and bad system but will not rule out well fitted systems, the types that are produced by your software or other machine learning software, and it appears that you are relying on a method or modification thereof that does not work.

If you know of a method that will guarantee that a system your software produces will work in unseen data in forward mode please let us know. You do not because nobody does. You reduce the data-mining bias inherent in bad system and you retain the best fitted systems because the MC, no matter how it is modified, cannot reject those.

I have been clear so please do not accuse me of being vague or just criticizing your work. You basically have nothing and you know that.

Interesting discussion–especially since it sounds like Bob is trying to expose another fraud, of which the financial landscape is littered. Unfortunately, while I am no dummy, I do not have a Ph.D. in mathematics or statistics. This leaves the specific points of debate somewhat over my head. I’d be interested in some clarification of those specific points if the commenters feel it worth their time to do so.

Very interesting discussion! It’s a pity that it has become so personal. I think datasnooping is one of the biggest problem with backtesting and should be discussed more.

An argument in favor of David is that Jaffray Woodriff (one of the most successful hedge managers) seem to do similar things: see p.152 of Schwager’s Hedge Fund Market Wizard. Here Woodriff explains that he looks at the difference in outcomes between the model on the real data and on the data with random disturbances. That seems very similar to your approach, isn’t David?

Also there is an interesting recent paper at SSRN (id=2308659), which seems in favor of Bob’s argument.

[…] Predictive-Model Based Trading Systems, Part 1 | System Trader … […]

Bob, what do you propose as an alternative to the MC method?

How can we improve the processes proposed in this text?

Although I do not hold a PhD in statistics, I have studied this topic fairly deeply. I also have not read Aronson and Master’s TSSB book (yet) but have read what Masters had earlier published regarding the use of MCP to evaluate trading systems as well as Aronson’s EBTA.

I read the fxhackers blog post per the link that Bob posted in the comment. Aronson is absolutely correct that Bob doesn’t seem to understand how Master’s MCP tests actually work. That blog post Bob suggested talks generically about Monte Carlo methods. Unfortunately the meaning of the term Monte Carlo is overloaded. Per wikipedia: “Monte Carlo methods are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results.” This is known in statistics as resampling. There are many methods of resampling including bootstrap, jacknife, and permutation.

The method discussed at fxhackers is actually a simple bootstrap using a set of trade results from a back-test. The term Monte Carlo in this case simply means random re-ordering of individual trade results. This method is the most naive use of resampling applied to trading system evaluation there is. So if that is what Bob believes that Master’s MCP tests are doing, I get his criticism.

But Master’s MCP tests are much different than a naive application of random re-ordering applied to trade results. It is an unfortunate consequence of the blog and comment format that discussions often degrade into arguing over generalities, especially when someone arguing has clearly not even familiarized himself with the content over which he argues. MCP tests are powerful where they fit the application.

With that said, I do believe the MCP principle is very limited in its applicability to a real trading system (at least what I’ve read thus far. My criticism is that (as Masters readily admits) the trading system must fit strict design requirements. Specifically, I’ve yet to see a resampling application for a trading system in the context of a realistic portfolio (multiple instruments, margin limits, position sizing limits, etc). I’d be very happily proven wrong on this comment. Perhaps the TSSB book provides improvements to earlier work and as I said, I need to read that.