Reproducibility crisis – Removing miracles from science
Reproducibility is a foundational principle of scientific research, with great care taken over reduction in experimental variables and documentation of methods. However, a 2018 Nature survey reveals that all does not seem to be well, with 86% of respondents acknowledging a crisis in their scientific field. Dominique-Laurent Courturier (1) cites a couple of publications that estimate at the very most only 37% of studies are reproducible.
We were lucky enough to be invited to attend and speak at last month’s Cancer Research UK (CRUK) symposium at Cambridge Institute on Reproducibility of in-vivo Data. We attended talks throughout the day and couldn’t help but witness the overwhelming desire and passion for consistent, reproducible research.
One particularly striking talk which brought this home was from Nature Journal’s, Barbara Marte (2). Barbara highlighted that variables as small as the veracity at which a test tube is shaken could have significant impact of the reproducibility of a study. Barbara spoke about the lengths at which Nature is now going to to validate and interrogate raw data sets alongside the introduction of a much more comprehensive submission check list, all in a drive to improve documentation of methods.
Furthermore, Florian Markowetz from CRUK reiterated this in a thoroughly entertaining but cuttingly sharp talk ‘5 selfish reasons to work reproducibly’ (3). Taking a look at the personal and community impact of working in an unreproducible manor, he surmised reproducible data helps write papers; people review papers; enables continuity between research operators; builds your reputation; and ultimately, promotes an “open research” culture. More on Florian’s point of view can be seen via this video.
One example Florian cites from his own research highlights the importance of reproducibility;
“Even smaller disasters can be embarrassing. Here is an example from my own research. Our experimental collaboration partners were validating a pathway model that we had generated computationally. When writing the paper, however, we hit a crucial roadblock: no matter how hard we tried, we could not reproduce our initial pathway model. Maybe the data had changed, maybe the code was different, or maybe we just couldn’t remember the parameter settings of our method correctly. Had we published this result, we would not have been able to demonstrate how the validated hypothesis was generated from the initial data. We would have published a miracle.” (3)
As 3D data and measurement experts we understand the impact even the smallest inaccuracy and variable can have on the outcome of a result, particularly when large data sets captured on multiple days and over multiple sites are involved. The importance of delivering a consistent interoperable experience and delivery of clean, and consistent data collection that can form the basis of traceable and robust analysis and evaluation cannot be understated. As Florian put it – “there are not and should not be any miracles in science”.
This challenge is the exact reason Fuel3D has partnered with leading in-vivo scientists and organisations across the industry to co-develop a new method for monitoring and capturing subcutaneous tumour growth indicators. Using the latest 3D imaging and reconstruction technology and machine learning, BioVolume® is now in evaluation with many partner organisations and results are positive and continually improving against the ordinary (and often unreproducible) manual calliper measurement methods, providing reproducible results.
(1) Laurent-couturier D – Senior Statistician CRUK Cambridge Institute www.cruk.cam.ac.uk/users/dominique-laurent-couturier
(2) Marte B. 2018. Reproducibility at Nature Journals. Cancer Research UK, Cambridge Institute.
(3) Markowetz, F. 2015. 5 selfish reasons to work reproducibly. Genome Biology, Springer Nature.