If an Experiment Fails in a Forest, Does Anyone Hear?

There are many reasons why Open Science is a good thing. For some it’s a moral argument that stresses sharing the results of (usually publicly funded) scientific research with society, preventing fraud through transparency, and benefiting teaching through the use of open materials. Others see the growing complexity and challenges of science as demanding collaboration; so that larger teams with their wider expertise can be brought to bear. Clearly there are personal benefits too; as Steve Lawrence has shown there is a correlation between sharing the results of research and the number of paper citations. Many innovators and entrepreneurs are also fond of Open Science because sharing technology can accelerate the innovation process and empower small business by reducing intellectual rights barriers. And there are a lot of us that just like to have fun–the communities and relationships that form in an open environment make the hard work of science that much more enjoyable.

While I agree with all of these sentiments, they miss the crucial issue: reproducibility. It was for good reason that the Royal Society was formed in 1660 with the militant motto “Nullius in verba” or rendered in English “take nobody’s word for it.” Once the scientific method was formalized and practiced by these and other pioneers we began to benefit from the power of science. The early scientists (actually Natural Philosophers) realized that an understanding of physical reality was based on the practice of objectively performing “experiments” and repeatedly reproducing the same results. Only then could something be called truth, and incorporated into our foundational knowledge base.

I sometimes wonder whether the early scientists were responding to a time of superstition; I can’t help but think of the rather humorous comedy skit from Monty Python. In the famous “How to Tell a Witch” the reasoning behind the determination as to whether an accused woman is a witch is something to behold, and ends up testing whether the purported witch weighs the same as a duck. It’s easy to see how superstition and emotion can produce a faulty decision about physical reality; yet using erroneous facts produce similar results. While hilarious when Monty Python plays it, unfortunately there is anecdotal evidence that for many people such faulty “reasoning” processes are alive and well. If you think scientists are immune, consider the recent study that found that 90% of published papers in preclinical cancer research describe work that is not reproducible, and therefore wrong. Such people never learned or forgot about the importance of reproducibility, with the result that we are developing therapies and pharmaceuticals on shifting ground.

I don’t think that many of us in the open source world became involved thinking that we were following in the footsteps of hallowed scientists. For most of us, the reasons articulated in the first paragraph were enough. Yet very quickly we realized that without reproducible results, i.e., testing, we were doomed to build on unstable foundations. So in the end the essential requirement that our software be built on “truth” demanded that we test constantly to ensure that our results were repeatable no matter what happened to the underlying platform, data, and algorithms. It is for this very reason that we are proponents of Open Access journals like the Insight Journal, and use tools like CMake, CTest and CDash at the heart of our software development process. In this way, when an experiment (test) fails in the forest we hear it, and take the necessary steps to maintain the integrity of our foundational reality (and to the benefit of our users who build on it).

Once the imperative of reproducibility is accepted, all of the other open practices follow. Without Open Access publications to describe and explain the experimental process, Open Data to provide controlled input, and Open Source to rerun computations and analysis, it is not possible to reliably reproduce experiments. So to my way of thinking, if you are a technologist then there is no choice but to practice Open Science. Anything else is tantamount to arguing that a witch weighs the same as a duck.

7 Responses to If an Experiment Fails in a Forest, Does Anyone Hear?

  1. Pingback: Luis Ibanez

  2. Pingback: Lisa Avila

  3. Pingback: Luis Ibanez

  4. Pingback: Will Schroeder

  5. Pingback: Lisa Avila

  6. Pingback: Will Schroeder

  7. Pingback: Marcus Hanwell

Questions or comments are always welcome!