System Trader Success
Share It!

System Performance and Confidence Interval

When you review the performance of a trading system, how do you know it’s worth trading? How do you know it’s the right system for you? How confident are you that it will continue to profit in the future? When it comes to evaluating your trading system there are many factors to take into account. Some of them are obvious such as Net Profit and Risk-Per-Trade. Others may be a bit more unfamiliar such as Sharpe ratio or Profit Factor. This article is going to be the first article in several where I highlight a method or idea that can help you gauge the quality of a given trading system. In this article I would like to highlight a statistical based metric that can be used to help indicate the likelihood a given system will continue to generate profits in the future.

Many people simply look at the net profit of a trading system assuming a system with more profit must be the better system. This is often far from a good idea. More profit may also mean more risk, deeper drawdown or other compromises to achieve those higher results. When testing trading systems during the development process or reviewing a commercially available system before making a purchase, it is advised to have a few metrics on hand that will allow you to make a wise choice. There is no one single score that will give you the definitive answer. Furthermore, everyone has unique risk tolerances and expectations on what is considered tradable. Yet, we can make smarter choices than simply looking at net profit. Here is one method you should be aware of.

Confidence Interval

It’s easy to find a trading system that has a positive average profit of $100 and then conclude it could be profitable into the future. But is there a metric we can use to help us predict what might happen into the future? A complicated approach would be to use the Monte Carlo method, but not everyone has access to this, but we all have access to a simple calculator. By visiting a topic in statistics called Confidence Intervals (CI) we can obtain a hint at what’s possible and perhaps find weaknesses in our seemingly profitable trading system.

The average net profit of a trading system is simply the historical P&L for each trade over a given time period. Let’s imagine a trading system that has produced 60 trades. Some of the trades are winners and some are losers. We add together the total P&L for each individual trade and divide it by the number of trades – 60 in this case – and we get $100. Clearly this is well above zero so in the long run, this system appears profitable.

However, we also know that individual trades can be very different from our average profit per trade. Some trades produce much larger winning trades while others produce smaller winning trades. Still, other trades produce a range of losing trades. If we graph each trade’s P&L and then draw a line representing our average P&L we would see each individual trade falls around our mean value of $100. In other words, the P&L for any trade will vary around this mean value. We can measure this variation and use it to estimate the likelihood the system might remain profitable.

Statistically speaking, a trading system that exhibits a large standard deviation of profit per trade will have an increased chance of failing in the future. This is true even if the average mean is currently profitable. But what makes a standard deviation too large? This is explained below when we attempt to use our confidence interval to estimate a likely range of average P&L values into the future.

The green line is the average P&L. This line is plotted for each trade and we can see it’s angled downward showing a falling P&L as time goes by. Each blue plot is a single trade’s net P&L.

What we wish to do with our confidence interval is estimate, with 95% confidence, if our system will likely produce a negative average P&L into the future. In other words, is it likely our seemingly profitable trading system is based upon chance? We can estimate this with our CI formula.

CI = t * SD / squareroot( N )
CI = Confidence Interval
t = T-score (we estimate value to be 2 and the reasoning behind this is beyond this article)
SD = P&L Standard Deviation for all trades
N = number of trades in our sample

With our imaginary trading system we have a $100 average net profit and 60 trades in our sample. Please note that in order for this method to work, you must have a minimum of 60 trades in your sample. Let’s also state the standard deviation for all trades is $400. With this information we can compute our 95% confidence interval.

CI = 2 * $400 / squareroot( 60 )
CI = $800 / 7.746
CI = $103.28

For simplicity let’s round the confidence interval to the nearest dollar which is $103. What do we do with this value? We create a range or band around our average net profit value of $100 by both adding and subtracting the CI value.

upper band = Average Net Profit + CI = $203
lower band = Average Net profit – CI = -$3

We have now created a range of -$3 to $203 for our average net profit. What does this mean? Based on our calculation we have estimated with 95% confidence that our trading system’s average net profit could be as low as -$3 or has high as $203. The important number is the lower band because this represents a worse case situation. In our example, we have a negative value which indicates a losing system.

In short, our hypothetical trading system’s average net profit could be based upon chance and in the future could produce a negative P&L. Suddenly what seems like a solid system seems more shaky. Does this mean our trading system should be abandoned? Not necessarily.

In the case of the confidence interval there are two factors at play which are critical. Those values are the number of trades (N) and the value of the standard deviation between trades. Modification of the standard deviation can be achieved by altering the trading system logic. Modifying stops, targets and other trading rules will change the standard deviation value. The goal would be to tighten the variation of each trade to reduce the standard deviation. This in turn, would create a smaller CI. However, if you don’t want to modify the system or if you are unable to modify the system there is another way.

Our example system was based on 60 trades. This is really not a lot of trades. Let’s say we find more data to test our system and we get up to 100 trades. Let’s also pretend all the other performance factors stay the same. If we recalculate our CI value we now get a value of $80. This gives us a range of $20 and $180. In this case, we have a system which produces positive results. So, maybe before we make a judgment on a system that appears borderline we should collect more trades.

I should also point out that our imaginary trading system we are looking at has $30 deducted for each trade to account for slippage and commissions. So this negative effect is already factored into our CI calculation. If we did not take into account slippage and commissions during our back-testing we would have to deduct this from our final range which would give us -$10 and $150. The impact of commissions and slippage just puts us back into negative territory gain. But we have them accounted for in our back-tested results.

As you can see having enough data points (trades) can have a significant impact on the CI calculation. For system trading there are many reasons for having a large number of trades. Of course continuing to add more and more trades is not going to turn a losing system into a good system. The point here is sometimes you need to have more data before making an informed decision. If you have what you believe is a good system, yet you only have a few data points, the CI calculation may be warning you to get more trades in the test sample.

What Confidence Interval Does Not Tell Us

CI does not indicate if this system has been curve fitted. If we have a killer system with 1000 trades with a CI range of $100 – $200 that’s great. However, it’s pointless if the system is curve fitted to the historical data and there is no way our CI calculation can tell us. But even if we have a solid system that is not curve fitted to the historical data our CI calculations are no guarantee of success in the future. The markets are dynamic and changing and it’s possible the distribution of trades will change thus altering our average trade and standard deviation. In the end, even if our system looks great on paper, we believe it’s not curve fitted and our CI interval looks fantastic, our trading system could fail as soon as we trade it live. If this is the case, what is the point of all this testing and is it worth doing? The short answer is, yes.

In trading there is no guarnatee for future results – ever. The point of testing a system is not to prove how much money it will make in the future. The goal is to find reasons why not to trade it. The goal is to find weaknesses so we can address those weaknesses now before we have money on the line. Our job as professional system traders is to manage risk which means eliminating risky actions. That’s all.

By using CI we have another tool to find weaknesses and ultimately give us more confidence that a particular system will likely continue to success into the future.

About the Author Jeff Swanson

Jeff is the founder of System Trader Success - an inBox magazine dedicated to sharing great ideas and concepts from the world of automated trading systems. Read More Google

follow me on:

Leave a Comment:

8 comments
Add Your Reply