How to publish responsible reproducible research

Scientific publishing has changed significantly in recent years, while the reproducibility crisis has deepened. The focus used to be solely on the manuscript. Today, it is much more about the complete publication, which includes data, computer code and lab protocols, making it easier for scientists to reproduce published studies.

To help researchers handle the growing list of requirements and continue to publish high-quality and reproducible research, the Luxembourg Centre for Systems Biomedicine (LCSB), part of the University of Luxembourg, has formally established a pre-publication check (PPC). The process itself is like the checklist pilots go through before every flight. The PPC is composed of seven modules, each checking a different aspect of the publication: manuscript content, data, code, lab experiments, commercialisation, sustainability, and other institutional aspects.

• The evolving meaning of ‘corresponding authorship’ in research
• Great citations: how to avoid referencing questionable evidence
• How much should you publish?

Publishing research results in a reproducible way is a challenging undertaking. Here is advice that your institute can follow to achieve this ambitious goal.

Build a team

The LCSB has pioneered responsible reproducible research (R3) since the centre was founded in 2009. The aim of the R3 initiative is to raise research quality and increase the overall reproducibility of scientific results. Its foundation relies on applying well-known change-management principles in a multidisciplinary research environment, following a bottom-up approach.

An R3 initiative will lead to significant cultural shifts. This effort, however, requires continuous involvement and support from motivated and informed staff with the right attitude and mindset. These staff members already exist at your institute: look out for them! These are the team members who care about improving existing processes, are well versed in teaching others or who feel the need to help their peers. Another tip for building a team that works well together is to mix seasoned members who know the institute inside and out with new arrivals, who will bring a fresh point of view.

Serve pizza

The adoption of the R3 principles has worked well thanks to the dedicated team: early adopters benefited from tailored support within a welcoming environment. We served pizza and enjoyed the ride. Start by giving practical advice. Share best practices to ensure reproducibility, show how common tasks should be performed, and soon you will feel the need to formalise. This is where the creation of a comprehensive publication pipeline will prove useful. The R3 effort matured for several years until the formal PPC process helped the LCSB to “cross the chasm” in 2023, according to the teachings outlined in Geoffrey A. Moore’s Crossing the Chasm.

Team up with management

An initiative to push boundaries and build a culture of reproducibility in academic research cannot be implemented without the support of management. Despite being primarily a bottom-up change, having the buy-in of decision-makers at institutional levels is crucial. This buy-in is either natural or can be achieved via clear communication towards management, highlighting that the R3 initiative is an integral part of achieving scientific excellence and fulfilling institutional responsibilities. The support can come in many forms: defining and signing policies, promotion to the researchers and outreach, providing resources, or leading by example. In our case, the integration of the PPC in the overall strategy of an institute is key.

Coach the coach

Once the PPC is established, nominate and train coaches for each of the modules from the following areas: data stewards, scientific software development experts, lab-quality and safety officers, technology transfer officers, attorneys and research facilitators, as well as grant officers. Most of these will already be aware of the challenges at the institute and can flexibly work towards improving the research reproducibility. It is also beneficial to have a few software developers who might be able to implement a unique publications platform that the researchers can easily use.

Get close

Use the PPC to identify issues early. Formally, the PPC highlights the shortcomings of an upcoming publication, giving the authors an opportunity to address potential problems before submission. Eventually, the process might open issues that have gone undetected for years or point at skills that are missing. These can include the omission or lack of data-transfer agreements, basic quality tests or elemental scientific computing skills. Most prominently, the PPC can demonstrate that most flaws could have been addressed early on during the project planning phase.

Getting close to the researchers allows for better understanding of their issues and provides faster and more focused help. Institutes have several options in terms of addressing issues: repetitive mandatory training sessions (such as data-management training), tailored and dedicated training sessions (training on computer topics such as the source code-management system Git) or individual consulting. Individual approaches work particularly well and can deliver results efficiently and quickly, but these are also challenging to scale up. A dedicated R3 clinic can serve as a framework for the R3 team members to provide project-tailored help and advice to researchers.

Automate

Once the adoption gap (that is, the “chasm”) between early adopters and the rest of the organisation is crossed, the PPC can be scaled up across the whole institute. No matter how committed your team is, the workload will present a new challenge that will need to be addressed. The team can benefit from the help of “R3 champions”, which is what we call users who have mastered the aspects of responsible science and can help their colleagues.

More importantly, automate large parts of the PPC process. For instance, text plagiarism can be detected automatically using iThenticate and the scientific rigour can be evaluated with SciScore. Many other tools may be integrated into the PPC process. For instance, a fully automatic bot can scan all code submissions. Plans include using the service from Copyleaks to check for code plagiarism and providing reports generated by FOSS tools such as Fossology to ensure open-source licence compliance.

Finally, assign an automatically generated static website (a so-called frozen page), for which a digital object identifier (DOI) is coined, to each publication. That way, the DOI will be accessible forever and the institute will guarantee the up-to-date validity of any supplementary information on the frozen page, ensuring long-term reproducibility and sustainability of research.

Document FAIR-ly

A key element in publishing responsible reproducible research is to make easy-to-understand documentation accessible to everyone at the institute by following FAIR principles (findable, accessible, interoperable and reusable). At the LCSB, we name them the “how-to cards”: an online knowledge base that is jointly developed by many teams using Gitlab, with the additional benefit of flexibility since the time required for a new issue to have a solution documented by a new how-to card is minimal. The individual cards outline the steps needed to complete the tasks at hand quickly (such as how to properly license a piece of software). The cards provide a practical view of the more formal policies duly signed by management.

Adapt

The PPC is dynamic: publishers and funders define new requirements, and new tools, policies and processes are established almost monthly. Every research project is slightly different and the work style can also vary among researchers. Harmonisation and standardisation might be desirable but are often quite challenging to achieve and eventually detrimental to the academic diversity. Be agile and learn. The common resistance to change from researchers can be overcome with a creative solution. It leads you to find a new approach to tackle the issue: adapt!

Following this advice might only help you to get started. Formalising, developing and perfecting the process is part of the basic scientific desire to strive for excellence and reproducibility. Yet clear and concise communication about this important change is key to success.

Laurent Heirendt is in charge of the responsible reproducible research (R3) team at the Luxembourg Centre for Systems Biomedicine (LCSB), an interdisciplinary research centre at the University of Luxembourg.

If you found this interesting and want advice and insight from academics and university staff delivered direct to your inbox each week, sign up for the Campus newsletter.