Have you ever seen a bad back-test?

Investment professionals have been jokingly asking that question for years, and the answer remains the same: of course not. That is because no one will likely visit your office to discuss a new product designed to be smart beta, strategic beta, or what I’ll call factor-based whose simulated history offers only mediocre performance.

Why? Because few would buy it.

Which brings us to how investment products are (often) made and how you can determine whether they are worth your clients’ money.

Factor-based products are often developed when asset managers examine historical data to try to determine what attributes of securities may have driven outperformance over time. Before a product is launched, a rules-based methodology may be implemented and applied to historical data as if the methodology had begun earlier, hence the term back-tested. What one person might consider research another might call data mining, and there can be a fine line between the two. But however you think about it, there is a difference between finding a random anomaly and identifying a viable rules-based strategy.

As a fun example of this, a few years ago my colleagues Joel Dickson and Chuck Thomas ran a hypothetical simulation that compared the performance of the S&P 500 Index with an equity portfolio that had an equally weighted combination of all stocks with tickers that began with S, M, A, R, or T. As the figure below shows, this simple, rules-based strategy did very well over a long period of time. However, let’s be honest, there is no sound reason to justify why it would be a good idea to pursue this strategy in the future.


Annualized return of S.M.A.R.T. beta strategy
from December 31, 1994, to October 31, 2013

Annualized return of S.M.A.R.T. beta strategy from December 31, 1994, to October 31, 2013

Source: Vanguard.

Note: The S.M.A.R.T. beta strategy is hypothetical in nature and does not represent the returns of any index or investment vehicle. It is constructed with equal-weighted components of all current securities in the S&P 500 whose tickers begin with the letters S,M,A,R,T. and rebalanced monthly.


While in most cases, it is hard to eliminate the risks of data mining entirely, there are a number of ways to help improve your confidence in the potential of a simple rules-based strategy to produce a return premium for a client in the future.

I invite you to download our one-page checklist for evaluating back-tested strategies, which identifies common biases that can occur when products are created. The questions in the checklist are ones you may want to ask any time you are considering a factor-based product.

These details are grouped under three overarching questions:


1. Is there an enduring, sensible rationale?

Consider whether there is a risk-based reason for the strategy. If there is, then there may be some intuition as to why the strategy may persist with rational asset pricing (in this scenario, no market inefficiency is needed). If the reason is that some investors believe other investors have made and will continue to make a similar mistake that creates an anomaly, then a sound reason should be given as to why some investors will continue to make that error and/or why other investors do not take full advantage of the mistake.


2. Does extensive evidence support the strategy?

In other words, what quantitative evidence supports this strategy generating a return premium? Geeks like me often call this robustness testing. This means taking a rigorous approach to looking at the historical data through advanced econometric techniques. There is a lot of jargon in this area of finance (terms such as out-of-sample tests, overfitting bias, and multiple testing bias); however, I’ll spare you from it here. In the checklist, we listed some questions that simplify the technical aspects, so you can ask managers/analysts the right questions.


3. Will the strategy work after all costs are considered?

This is often an underappreciated aspect of due diligence. Investors must believe these return premiums will survive real-world implementation costs (e.g., trading commissions, bid-ask spreads, management expenses, and taxes), which can really add up over time. The actual significance of these various effects can be influenced by numerous issues, such as the specific rules-based strategy, the way securities are weighted, the size of the potential investment, the rebalancing policy, the type of vehicle chosen for implementation (e.g., ETF or conventional mutual fund), and the circumstances of the client under consideration.


In the coming weeks or months, you will no doubt either see an advertisement about or hear a salesperson pitch a rules-based strategy. Just remember the importance of thoroughly evaluating whether the efficacy of a strategy that may have worked in the past is real or just random.


I would like to thank my colleague Tom Paradise for his contributions to this blog post.