This year’s top 10 stories included a peek into AustralianSuper’s international equities build out in London, AMP’s move to slash employee benefits, and plenty of hard-hitting analysis of the issues that matter in institutional investment. But the real story is how readers helped shape all of that coverage.
Orbis’ Stuart Place is riding from Melbourne to the Moon and Back to fund a treatment for the “monster of a disease” that his youngest son was born with. The investment industry is rallying behind him.
Back-test backlash: why financial research could be blowing smoke
A new academic paper has labeled most investment strategy research as statistically flawed, equating finance industry back-tested analyses with the now-debunked historical medical studies funded by ‘big tobacco’.
In the study titled ‘Finance is not excused: why finance should not flout basic principles of statistics‘, authors David Bailey and Marcos de Prado, argue that the majority of financial research based on interrogating historical data to discern ‘optimal’ investment strategies has served up mostly “false discoveries”.
“With this ‘optimal’ design in hand, [financial researchers] tout the potential return that an investment based on this design is likely to deliver, based on its simulated performance on historical data (backtest). However, in all too many cases, such investments deliver only disappointing performance when actually fielded,” the report says.
“… The sobering consequence is that a significant portion of the models, funds and strategies employed in the investment world, including many of those marketed to individual investors, may be merely statistical mirages.”
Bailey and de Prado – respectively, University of California research fellow, and, Abu Dhabi Investment Authority global head quantitative research & development – suggest that three features of financial research render it prone to error, including:
- the low probability of finding a “profitable investment strategy” in a highly competitive market;
- the short shelf-life of any effective strategies (as they are quickly arbitraged away); and,
- the fact that “unlike in the natural sciences, it is rarely possible to verify statistical findings through controlled experiments”.
As well as questioning some of the standard investment skill performance measures such as the Sharpe Ratio (the authors propose a ‘deflated’ version of the metric), the study skewers financial researchers for “overfitting” back-tested results to derive ultimately disappointing real-world strategies.
For example, the report says back-test pre-launch studies of the many thousands of exchange-traded funds (ETFs) released between 1994 and 2014 predicted average index outperformance of 5 per cent.
“This strong performance contrasts with average annual excess returns of approx. 0% out-of-sample,” the study says. “… Such disappointing behavior is entirely consistent with a design process that involves extensive computer exploration of index parameters and selecting only the ‘optimal’ parameters for an index fund subsequently fielded in the financial markets.”
Bailey and de Prado also refute counter-claims that the “existence of a replication crisis” in financial research has been over-stated.
The report says some critics suggest that “concerns with backtest overfitting are overblown” or strategies such as factor-investing might be immune from the statistical errors.
“We do not concur,” the study says. “Rather, the preponderance of poor out-of-sample performance points to a pervasive problem in the field.”
According to the report, the financial research industry has produced thousands of studies that “promote dubious investment strategies” based on statistically erroneous methods.
“In some cases, such methodological error could be attributed to negligence, but in other cases it responds to conflicts of interest,” the paper says.
And in a world where no-one fails a back-test “today’s academic finance exhibits some resemblance with medicine’s predicament during the 1950-2000 period, when Big Tobacco paid for thousands of studies in support of their bottom line”.
“Unlike finance, medical journals today impose strict controls for multiple testing,” the study says. “Academic finance’s denial of its replication crisis risks its branding as a pseudoscience.”
The paper warns statisticians considering a career in finance should take care to choose employers committed to using only “rigorous, objective statistical methodologies”.
“… ask yourself whether you want to be part of the solution, or part of the problem,” the study says.
