Since the emergence of “Modern Portfolio Theory” and the “Capital Asset Pricing Model” in the late 1960s, institutional investors have taken a quantitatively driven approach to portfolio construction, looking to create portfolio diversification and obtain better risk-adjusted returns by balancing their asset-class exposures. This journey has seen several important advancements in thinking about how to optimally achieve desired results.
Funds that want to take the total portfolio approach first need to get the total portfolio view. To do that they not only need data – and lots of it – but a rock-solid understanding of exactly how they’re going to use it.
Back-test backlash: why financial research could be blowing smoke
A new academic paper has labeled most investment strategy research as statistically flawed, equating finance industry back-tested analyses with the now-debunked historical medical studies funded by ‘big tobacco’.
In the study titled ‘Finance is not excused: why finance should not flout basic principles of statistics‘, authors David Bailey and Marcos de Prado, argue that the majority of financial research based on interrogating historical data to discern ‘optimal’ investment strategies has served up mostly “false discoveries”.
“With this ‘optimal’ design in hand, [financial researchers] tout the potential return that an investment based on this design is likely to deliver, based on its simulated performance on historical data (backtest). However, in all too many cases, such investments deliver only disappointing performance when actually fielded,” the report says.
“… The sobering consequence is that a significant portion of the models, funds and strategies employed in the investment world, including many of those marketed to individual investors, may be merely statistical mirages.”
Bailey and de Prado – respectively, University of California research fellow, and, Abu Dhabi Investment Authority global head quantitative research & development – suggest that three features of financial research render it prone to error, including:
- the low probability of finding a “profitable investment strategy” in a highly competitive market;
- the short shelf-life of any effective strategies (as they are quickly arbitraged away); and,
- the fact that “unlike in the natural sciences, it is rarely possible to verify statistical findings through controlled experiments”.
As well as questioning some of the standard investment skill performance measures such as the Sharpe Ratio (the authors propose a ‘deflated’ version of the metric), the study skewers financial researchers for “overfitting” back-tested results to derive ultimately disappointing real-world strategies.
For example, the report says back-test pre-launch studies of the many thousands of exchange-traded funds (ETFs) released between 1994 and 2014 predicted average index outperformance of 5 per cent.
“This strong performance contrasts with average annual excess returns of approx. 0% out-of-sample,” the study says. “… Such disappointing behavior is entirely consistent with a design process that involves extensive computer exploration of index parameters and selecting only the ‘optimal’ parameters for an index fund subsequently fielded in the financial markets.”
Bailey and de Prado also refute counter-claims that the “existence of a replication crisis” in financial research has been over-stated.
The report says some critics suggest that “concerns with backtest overfitting are overblown” or strategies such as factor-investing might be immune from the statistical errors.
“We do not concur,” the study says. “Rather, the preponderance of poor out-of-sample performance points to a pervasive problem in the field.”
According to the report, the financial research industry has produced thousands of studies that “promote dubious investment strategies” based on statistically erroneous methods.
“In some cases, such methodological error could be attributed to negligence, but in other cases it responds to conflicts of interest,” the paper says.
And in a world where no-one fails a back-test “today’s academic finance exhibits some resemblance with medicine’s predicament during the 1950-2000 period, when Big Tobacco paid for thousands of studies in support of their bottom line”.
“Unlike finance, medical journals today impose strict controls for multiple testing,” the study says. “Academic finance’s denial of its replication crisis risks its branding as a pseudoscience.”
The paper warns statisticians considering a career in finance should take care to choose employers committed to using only “rigorous, objective statistical methodologies”.
“… ask yourself whether you want to be part of the solution, or part of the problem,” the study says.
