Dr. Patrick Welton Founder and Chief Investment Officer [email protected]
Regrettably, only a fraction of published medical research is reproducible by other investigators. This is no surprise to those of us who contribute or follow medical advancements. Hundreds of reviews have documented tens of thousands of examples. From cancer biology to clinical treatment tools to any study involving survey techniques and more, only a fraction of research will stand the test of time by proving true in others’ hands.
Investors face a similar challenge. This article from Bloomberg, “The Finance Research Crisis is All Too Real,” highlights the same pattern in financial research. Though causality varies, the net result is that a large percentage of what investors read will not turn out to be true.
How can this be? More probingly, why should this be?
While incompetency undermines many papers and nefarious incentives spawn a select few, these do not explain the pervasiveness of irreproducible research results.
Difficulty does. Discovery is hard. And the discovery of underlying truths in domains with high variability like medicine and financial markets is harder yet.
The design, controls, measurements, and statistical applications are challenging and often climb exponential obstacles of complexity and cost. When I was teaching, I often shared with my residents and postdocs that many answers in medicine lay beyond the aphoristic reach of finding the needle in a haystack. For some questions in cancer, the needle and haystack always seemed to be moving, scattering in swirling wind as one looked. If I were to extend this metaphor to financial research, without putting too fine a point on motivations, many needles look like gold when the person looking wants them to be gold.
The beauty of science reflects human curiosity and creativity. And the path to beauty can be washed out by a single lapse of process discipline and the way rutted by the interactions of human bias confronting underlying noise. As humans, we are neurologically wired for biases. Real-world phenomena are an admixture of truth and random noise. This intertwining makes objective discovery hard. I recommend Noise by Kahneman, Sibony, and Sunstein for those interested in learning more.
So why reproducibility? Because reproducibility is the crucible for testing objective truth. And real advancements are as equally true in the hands of the discoverer as they are in the hands of any other.
Each of us contributes to knowledge advancement by applying a core principle of science. Be skeptical.
Always question. Study the data. Review the assumptions. Follow the published logic. Mind any gaps. If any of us read only the summary or the conclusion, we indiscernibly consume information and misinformation alike, giving up our small contribution to knowledge advancement.
