NIH ponders on Reproducibility Rules

In a recent blog post in Nature, Meredith Wadman shared details on the NIH ongoing initiatives to foster the verification of reproducibility in the research activities funded by the agency.

“Nullius in Verba”
“Take Nobody’s word for it”

“It is an expression of the determination of Fellows of the Royal Society
to withstand the domination of authority
and to verify all statements by an appeal to facts determined by experiment.”


After having hosted two workshops on the issues reproducibility in 2012, the National Institutes of Health (NIH) are just beginning to take active measures to restore the proper practice of scientific research.

Lawrence Tabak, the agency’s principal deputy director, stated

“There is certainly sufficient information now that the NIH
feels it’s appropriate to look at this at a central-agency level,”

Wadman reports that

This summer, Tabak and other senior NIH officials, including Story Landis, director at the neurology institute, and Harold Varmus, director at the cancer institute, are assessing input gathered from the directors of the agency’s 27 institutes and centres. They will then confer with NIH director Francis Collins, who will decide what steps to take.

This is on the context of recent findings, that Wadman captures in the following list:

  • September 2011  Bayer HealthCare finds inconsistencies between in-house and published data in almost two-thirds of 67 projects.
  • March 2012  Amgen publication shows that the findings from only 11% of 53 landmark papers can be replicated by company scientists.
  • January 2013  Elizabeth Iorns, chief executive of Science Exchange, asks more than 22,000 study authors if they want her company to arrange verification of their experiments; nearly 2,000 say yes.
  • May 2013  A survey at the MD Anderson Cancer Center finds that more than half of its respondents have tried and failed to reproduce published data.
  • July 2013  Science Exchange launches a verification programme for commercially sold antibodies.


Tabak suggested:

“If the premise isn’t validatable,
then we’re done;
it doesn’t matter how well you wrote the grant,”

Wadman states also that:

Agency officials are also considering a requirement
that independent labs validate the results of important
preclinical studies as a condition of receiving grant funding.

There is of course, the question of who will pay for reproducibility and how much would that be. This is what makes so significant that finally the conversation has escalated to the level of the largest US Federal funding agency.

It has been estimated that the cost of a reproducibility study will be in the range of $25,000. This is about 5% of the original cost of doing the research, and when put in context, it is a small cost to pay for ensuring that the initial social investment in scientific research truly leads to results that stand to inspection.

Scientific reproducibility is therefore emerging as a Market Service, whose customers are any of the R&D organizations that take as input the results of basic research funded by public agencies.

The pioneer in this field is Science Exchange,

a start-up that has taken a free-market approach to bring the benefits of reproducible research to those who actually care about the research being more than a flashy paper published in journals where reviewers do not make any effort for independent verification.

Other key players in the space of Reproducible Research are

  • Run My Code: a site that allows researchers to replicate scientific results in order to check their validity and robustness.
  • Figshare: a place where data, negative results and true verification materials are shared publicly.
  • PLoS ONE: the progressive mega-journal that is require software-intensive papers to include source code under open source licenses.
  • Mendeley: The platform for managing and sharing research papers.

These progressive organizations are taking upon themselves to restore the honest practice of the scientific method, that is anchored on the practice of reproducibility verification.

( under CC SA 2.0 License)

The Renaissance of True Science is now Unstoppable.

3 Responses to NIH ponders on Reproducibility Rules

  1. Pingback: Bill Lorensen

  2. Pingback: Bill Lorensen

  3. Pingback: Luis Ibanez

Questions or comments are always welcome!