How To Spot A Fake

One of the issues that comes up regularly is how, as an investor or other interested party, one can protect oneself from unscrupulous scam artists posing as professional traders or money managers. This is a particular problem on web sites featuring trader forums, where individuals with unverified track records claiming stellar trading histories use their purported trading “prowess” to try to impress and intimidate other participants, usually impressionable newbies. The purpose of this post is to provide some guidance to help investors, traders and other fellow travelers sort the wheat from the chaff. We’ll be doing some forensic analysis on the track record for a strategy in NG futures that one such character recently posted in one of these forums, as a classic example of the kind of fakery I am describing.

One thing you should understand about scam artists operating on forums, is that they don’t work alone: usually they have a bunch of groupies who will shill for them at every opportunity and who will try to shout down any investigative questioning. Don’t be deterred. These know-it-alls are usually just ignorant dupes, who understand no more about trading than the scam artist. They may just as easily be fellow-scam artists themselves.

THE FIRST BIG RED FLAG: UNWILLINGNESS TO PRODUCE A TRACK RECORD

Anyone claiming to be a CTA or professional money manager (or whose shills claim he is one) has to have a track record that is freely available in the public domain. So how does a scam artist overcome a challenge to produce it? He will claim that he “can’t advertise”, or make some other, similar excuse. Don’t accept that at face value. Ask him to PM it to you. If he won’t, there’s already a high probability he’s a con artist.

THE SECOND BIG RED FLAG: CURVE FITTING

Let’s say our suspect meets the challenge and produces a track record. Ideally this will be an audited P&L statement, but let’s assume for the purposes of this discussion that he produces something along the lines of the Performance Reports produced by a product like Tradestation or MultiCharts, i.e. we are dealing with a simulated back-test.

If your suspect produces a back-test, you can be pretty sure it’s going to look good – otherwise he wouldn’t produce it. The task now is to dig into those reports to spot the red flags that give clues as to whether it might be fake.

Now of course any trading system is going to make assumptions – about fill rates, slippage, commissions, capacity etc. All that is fine, as long as the assumptions are clearly stated. You might want to challenge any or all of the assumptions, and the trader may disagree with you about some or all of them. That’s perfectly ok – it’s an honest, open discussion about a set of investment assumptions that have been revealed at the outset.

But here is what is NOT ok: any opacity about which data was used to build the trading model and which data was used to test it. The former, the in-sample (IS) data set, used to construct the model, must be entirely separate and distinct from the out-of-sample (OOS) data set. It is trivially easy using a tool like Tradestation to produce a trading system that shows stellar results in-sample, but which will immediately crash and burn when it is used in live trading. This is known as curve-fitting. And it’s by far the most common method by which scam artists try to dupe investors.

In order to demonstrate the robustness of the system prior to risking real money, a genuine trader will test his system OOS and show you the results. What you are looking for ideally is congruity between the IS and OOS results. Now by congruity, I don’t mean that they should be identical. Far from it – markets evolve and strategy performance will vary over time. But what you are hoping is that the key performance metrics in the OOS and IS periods, such as annual returns, Sharpe ratio, PNL per contract, profit ratio and win rate, will be comparable. At the very least, you would like to be able to identify some portion of the IS data set for which the strategy performance characteristics are similar to those in the OOS period.

Any – I mean ANY – ambiguity or lack of clarity about which data was used to build the model and which was used for OOS testing is a HUGE red flag. Chances are, your scam artist is already trying to fudge the issue that he curve-fitted the system. This was the case in the recent forum post we are using as a test case. The trader made no attempt whatsoever to clarify which data was used for model development and which for testing. Immediately, I was suspicious and began looking for other evidence of curve fitting. It didn’t take me long to find it.

THE THIRD BIG RED FLAG: THE EQUITY CURVE

The first item I turned to in the performance reports was the equity curve and I immediately spotted two rather large clues that I was dealing with a fake.

The first clue was the large sign on the chart labelled “live start date”. What does this mean? This is a back-test, so all of the results are theoretical, including those after the supposed “live start date” sometime in 2013. What the faker is trying to do is imply the part of the equity curve shown after that date indicates actual performance results. He doesn’t actually claim this, so he has plausible deniability if you call him on it (“I said it was just a back test”). But he hopes that you won’t, and that, by default, you’ll accept these results are real. But they aren’t.

The second clue of fakery is much more important: the equity curve itself. When someone shows you an equity curve like the one reported by this trader, rising in a straight line from the lower left to upper right quadrants, you can be 99% confident that you are dealing with a fake. You see, in finance there are almost never any straight lines. They are as rare as unicorns. Especially when it comes to strategy performance. They only time you will EVER see an equity curve like this is when you are looking at the equity curve of (i) a high frequency market making trading system or (ii) a fake, produced by curve fitting a strategy to the ENTIRE data set. And this strategy was not high frequency – as we shall see, it operated on 15 minute bars, holding positions overnight.

EC-Chart

THE FOURTH BIG RED FLAG: GOD’s EQUITY CURVE

I said that straight line equity curves were extremely rare. In fact, even God’s equity curve isn’t often a straight line. What does that mean?

Suppose you had a strategy that could predict with 100% accuracy whether the market would go up or down over the next bar (whether you are using daily bars, or 15 minute bars, as in our example). The system would buy (or hold) when the market was forecast to rise, and sell when the market was predicted to fall. What would the performance of such a perfect system look like? Pretty stellar, obviously. And most people would guess that the system’s equity curve would be a straight line, or maybe even exponential in shape. In fact, that’s typically not the case. God’s equity curve will be sloped and kinked, just like any other equity curve. And if your suspect’s equity curve is real, it should show some commonality with God’s equity curve, by which I mean it should show changes in slope and level that reflect those seen in the perfect equity curve.

What does God’s Equity Curve look like in NG futures?

Gods-EC

As you can see it’s not straight. In fact it’s concave. So a REAL equity curve should have similar characteristics, like this one, for example:

NG-EC

As you can see, the equity curve of the real trading system track’s God’s Equity Curve, albeit at a much lower level. It’s concave, with an upswing during the final few months of trading, just like God’s. That’s a good sign that the strategy back-test is very likely genuine (which it is – I produced it).

Why is Gods’ Equity Curve the shape it is? The answer will vary from market to market. In the case of NG, the suggestion is that the market is becoming more efficient: simple trading strategies based on technical indicators work less well than they did five years ago. We have seen something very similar in F/X markets. During the 1970’s and 1980’s when Soros was active in the field, simple strategies like moving average crossovers made great returns, but these entirely dissipated in the 1990’s, with the advent of widely available computing power.

THE FIFTH BIG RED FLAG: THE SHILL SHOUTDOWN

When I posted my analysis, which clearly indicated fakery by this well known forum participant, I was immediately flamed by one of his supporters who shouted something to the effect that (i) everyone knows that the downward slope of God’s Equity Curve was caused by volatility and (ii) the star trader, unlike God, or me, knows about position sizing.

This attempt at misdirection in the face of awkward facts is a classic sign of fakery. What distinguishes the shill post is:

(i) Immediacy – clearly no attempt has been made to evaluate the argument or analysis. The shill simply attempts to drown out the critic with a lot of noise, as quickly as possible.

(ii) Plausibility – shills will throw around terms that lend plausibility to their objection, but which after a moment’s reflection are entirely irrelevant or, as in this case, detrimental to their own cause.

(iii) Invective – the more intemperate the post, the more likely the shill is simply trying to provide cover for the faker.

So let’s take a moment to dispose of the plausible sounding objections posted by the shill in this example. I am going to take it as read that everyone understands that trading profitability is positively correlated with volatility. There is a huge amount of empirical research supporting that finding, but to keep it simple we can appeal to one of the cornerstones of modern finance: risk and return. The higher the volatility, i.e. the greater the risk, the greater the return traders and investors in the markets will require on their capital. This is a principle of modern financial theory that even a graduate of the Scranton college of fine art should be expected to appreciate.

So what’s the story with NG volatility? You can see the time series of NG volatility in the chart below. One feature stands out above all others: the upward slope of the curve. NG volatility has RISEN over the sample period from 2008 to 2014. Consequently, returns from trading NG futures should also have RISEN rather than fallen. One thing we can say for sure, whatever caused the concave shape in God’s Equity Curve in NG futures, was NOT volatility!

NG-Volatility

Turning to the shill’s next plausible sounding, but dubious “explanation”, position sizing: this really is completely irrelevant. Because, as we shall see from an examination of the performance report, the track record was created by trading a constant one-lot! So this was just an attempt to sound “sophisticated” by someone trying to misdirect the reader away from the increasingly obvious evidence of fakery.

THE SIXTH BIG RED FLAG: LOW DRAWDOWNS AND OVERNIGHT GAP RISK

One of the highly unusual features of our faker’s equity curve is its exceptional smoothness. Low volatility in the equity curve is, in and of itself, an indicator the track record results from curve fitting. But we can get even more insight by digging into the performance report, shown below.

Perf-1

 

Perf-2

As you can see from the second page of the report, the strategy holds positions for an average of 57 15-minute bars, equivalent to slightly over 14 hours. So this is a low frequency strategy that takes overnight risk. Now, as any trader will know, overnight gap risk in a product like NG can be very significant and likely to produce much larger drawdowns over a 5 year period than the $8,470 reported here.

The only other possible explanation is that the strategy is traded continuously through both day and night sessions. But this is not only itself improbable, it gives rise to another implausibility: liquidity in the overnight session is so poor that the strategy is unlikely to be able to trade more than 1-2 contracts, at most. This would be of little value to a CTA, or its customers, whatever the star trader’s protestations that his “clients are happy”.

There is no plausible way to resolve the disconnection between the low drawdown, overnight gap risk and market illiquidity. The most plausible explanation: the back-test is a curve fitting exercise.

THE SEVENTH AND FINAL BIG RED FLAG: INCONSISTENCY BETWEEN PERFORMANCE METRICS

As any experienced strategy developer knows, you can get some of the things you want, but you can never achieve all of them. Amongst the desirable features to be maximized are:

  • Profit factor
  • Average PNL per contract
  • Percentage win rate

There is a trade-off between the features. A high PNL per contract typically means you are trading less frequently, with longer hold periods, and consequently the percentage win rate tends to be lower. Alternatively, you can increase the win rate at the cost of lowering the average PNL per contract and/or the profit factor. And so on.

This strategy purports to have it all: a high average PNL per contract resulting from low frequency trading, coupled with a good percentage win rate of over 50% and profit factor. A win rate of much over 40% is highly unusual for a momentum strategy entering and exiting with market or stop orders – and its almost inconceivable for a strategy with a PNL per contract and profit factor as large as suggested here.

CONCLUSION

This back-test fails the sniff test on so many levels, I would rate the chance of it being real as less than 1 in 1000. The final, conclusive proof of fakery is that the “star trader” responsible for producing the report was unable and/or unwilling to attempt to answer even a single one of the criticisms.

So, be warned. If you see forum members banding about track records like this one, you can be sure that they and their strategies are likely to be fake, and not to be trusted.

— Jonathan Kinlay from Quantitative Research and Trading.

About the Author Jonathan Kinlay

Dr Jonathan Kinlay is the Head of Quantitative Trading at Systematic Strategies, LLC, a systematic hedge fund that deploys high frequency trading strategies using news-based algorithms. Dr Kinlay, was the founder and General Partner of the Caissa Capital hedge fund, whose volatility arbitrage strategies were developed by Dr Kinlay’s investment research firm, Investment Analytics. Caissa, which managed $400M in assets, was ranked by FIMAT as the top performing fund in its class in 2004. Dr Kinlay went on to establish the Proteom Capital, whose statistical arbitrage strategies were based on pattern recognition techniques used in DNA sequencing. Dr Kinlay was formerly Global Head of Model Review at the US investment bank Bear Stearns. Dr Kinlay holds a PhD in economics and has held positions on the faculty at New York University Stern School of Business, Carnegie Mellon and Reading Universities. Dr Kinlay is a regular conference speaker and writer on investment research, hedge fund investing and quantitative finance. Kinlay was a member of England’s chess team that won gold in the World Student Olympiad in Mexico in 1978. He is the son of Fleet Street editor James Kinlay and father of British actress Antonia Kinlay.

  • BlueHorseshoe says:

    Thanks for an interesting article, Jonathan.

    I think it’s worth keeping in mind that sellers of systems or hedge fund start-ups who are absolutely genuine, can also face difficulties that may easily be confused with fakery.

    For example, suppose I am trading a successful strategy in a $10 account. IS and OOS, it has good metrics, and returns around 10% per year. In my small account, however, annual commissions of $250 eat up around a quarter of that return.

    If I come to you wanting you to invest $500k in the strategy, then how do I present this? Obviously an economy of scale kicks in – my brokerage fees on a $500k account mightn’t be much more than the same $250, so they would no longer make much of a dent in the 10% return.

    If I want you and a bunch of others to invest $10 million each, then brokerage arrangements are going to be a completely different ball-game again…

    So how should I present this to you, the prospective investor?

    If I present correct and audited accounts then I only appear to return 7.5%. If I ignore commissions or estimate realistic ones for your investment then you may well accuse me of fakery. If I try and explain all of the above to you honestly, then you may just get confused and lose interest.

    Interested to hear your thoughts on this.

    • Its fine to ignore fees and trading costs, as long as you make that clear in the performance report. Generally speaking, however, most investors tend to prefer to see returns net of fees and costs.

  • James says:

    I’m not saying the results that you were discussing are genuine. I don’t know. However, the rate of change of a perfect equity curve is directly related to volatility. If you have a very good trading system then a straightish equity curve can be produced by taking volatility into account.

    • It’s not clear precisely how you take volatility into account. In general, volatility is time-varying, as are strategy risk-adjusted returns. So the equity curve tends to be non-linear.

  • Joe says:

    All these are good but one could argue (not me, because I think there are not any objective ways of judging in this area) that they also raise questions about some claims made on your website:

    E-mini system straight equity curve:

    http://jonathankinlay.com/index.php/2014/06/how-not-to-develop-gp-strategies-a-cautionary-tale/

    Another example:

    http://jonathankinlay.com/index.php/2014/11/equity-curve-money-management/

    Most of the equity curves are straight lines with no drawdown:

    http://jonathankinlay.com/index.php/2014/11/building-systematic-strategies-a-new-approach/

    VIX straight line:

    http://jonathankinlay.com/index.php/2014/10/a-calendar-spread-strategy-in-vix-futures/

    In addition, your statement that

    “A high PNL per contract typically means you are trading less frequently, with longer hold periods, and consequently the percentage win rate tends to be lower. ”

    Obviously here you are referring to your own experience unless you could provide a mathematical justification of such a bold statement.

    • I have replied to you other points in a separate comment. As regards this remark:
      “A high PNL per contract typically means you are trading less frequently, with longer hold periods, and consequently the percentage win rate tends to be lower. ”

      I can offer an intuitive explanation, and a mathematical one. Intuitively, risk increases with the holding period (approximately with the square root of time). If you are holding for longer periods you risk greater, as evidenced by larger drawdowns. You would expect to be compensated for taking more risk by earning larger returns. So you will have fewer trades, with larger drawdowns, but with larger average return.

      Mathematically, assuming your strategy behaves like a Geometric Brownian Motion process with annual (positive!) drift mu and variance s^2, then the expected return of your strategy over k years is

      ER = k*mu + k(s^2)/2

      whereas the median return is

      MR = k*mu

      In other words the median returns declines over time relative to the mean return.

      So, over longer periods your win rate tends to decline relative to your average return by a factor k(s^2)/2.

      The higher the volatility s of your strategy, the more pronounced this effect is.

      So, for example, you can see this effect very clearly with 3X leveraged ETFs.

      • Joe says:

        I posted this at the wrong place. I hope the admin will remove the redundant post

        What you say here would be true if the variance was constant. But we know that in real markets variance is not constant. Therefore no conclusions of the sort you made can be made from those formulas in general, which BTW I think they may be problematic because even it’s been a long time since MBA I think that

        ER = k*mu – k(s^2)/2

        and not

        ER = k*mu + k(s^2)/2

        I would expect volatility to have an adverse effect on the percentage drift. Otherwise life would be beautiful, wouldn’t be?

  • “The only time you will EVER see an equity curve like this is when you are looking at the equity curve of (i) a high frequency market making trading system…”

    Which is what the post on the VIX spreading strategy was about.

    As regards the E-Mini and related equity curve strategy are concerned:
    (i) There are some significant drawdowns evident in the equity curve, unlike the track record for the fake strategy.

    (ii) in the original post in the Tradestation forums, I recall that I published the Easylanguage code so readers could verify the system for themselves.

    The focus of other article referred to was on the application of genetic programming in strategy development, something I have been involved in for the last 15 years. It was intended to discuss an approach to strategy development, rather than any specific strategy.

    • Joe says:

      Dear Sir, you wrote in your kind response to me:

      “The focus of other article referred to was on the application of genetic programming in strategy development, something I have been involved in for the last 15 years. It was intended to discuss an approach to strategy development, rather than any specific strategy.”

      However, in this present article you say that this kind of equity curves raise red flags. Either they do or they do not. In that article you never mentioned that this type of equity curve is suspicious. You wrote in the present article:

      “When someone shows you an equity curve like the one reported by this trader, rising in a straight line from the lower left to upper right quadrants, you can be 99% confident that you are dealing with a fake. You see, in finance there are almost never any straight lines. They are as rare as unicorns. ”

      So I assume that the genetic programming equity curve was a unicorn with 99% confidence based on your current article. What happened, have you changed your mind since?

      • Joe says:

        What you say here would be true if the variance was constant. But we know that in real markets variance is not constant. Therefore no conclusions of the sort you made can be made from those formulas in general, which BTW I think they may be problematic because even it’s been a long time since MBA I think that

        ER = k*mu – k(s^2)/2

        and not

        ER = k*mu + k(s^2)/2

        I would expect volatility to have an adverse effect on the percentage drift. Otherwise life would be beautiful, wouldn’t be?

  • Alon says:

    “. During the 1970’s and 1980’s when Soros was active in the field, simple strategies like moving average crossovers made great returns, but these entirely dissipated in the 1990’s, with the advent of widely available computing power.”

    They still do. I think the author should now that. This is low quality article. The author appears angry and his statements are loaded. I have seen better article here. It is better to read about what to do right not about what other people do wrong. This is negative energy.

    • rob says:

      I agree. Also consider that he has the same “Fake” equity curves on his site.

      • Alon says:

        Rob well said. I just spotted a fake and unfortunately it is from the author’s latest blog:

        http://jonathankinlay.com/index.php/2015/03/combining-momentum-mean-reversion-strategies/

        Straight equity line except for a small random drawdown. Pure hindsight in action.

        Hi Jeff, this article is about what you can read in forums where people enter into fights about the results of some backtest. Good systems are always judged based on real performance and there is nothing more to say in this area. The problem here is that we have someone who shows plenty of straight equity curves in his blog but then attacks them in your blog. I have a problem to understand the motivation for that. Does he want to say that he is the only one who can do that and all the others are fakes?

    • John says:

      Nope. The author is 100% correct here. Simple MA crossovers, breakout strategies, etc. worked great in the 1970s/1980s. It’s not just Soros–look at all the trend followers who made millions back then. As computerized trading became more sophisticated in the early 1990s, markets became much more choppy. Many systems stopped working in that period.

      For the record, I think trend-following is still a viable strategy, but you need more sophisticated approaches now.

      If I’m wrong, please show me some highly-successful, long-term MA crossovers that have worked since the early 90s.

      • Sam says:

        A 50/200 moving average in S&P 500 has generated high returns. All you need to do is backtest it. More importantly, Soros made macro beds. He never used trading system the way discussed in this article. Where did you get your information? Does Soros look like a “trading system” guy to you?

  • >