Congress pushes NIH to review reproducibility and ‘near misses’

The National Institutes of Health, the leading provider of basic research funding to US academia, is under a new round of pressure from Congress to improve its operations in multiple areas, including reproducibility and peer review.

As part of the NIH’s annual budget bill for the current fiscal year, lawmakers ordered actions led by a set-aside of at least $10 million (£8 million) for the agency to fund replication experiments on the agency’s “significant lines of research” and “proactively look for signs of academic fraud”.

Other areas that Congress ordered the NIH to examine and test out included new funding models such as lottery-style awards, improving the quality of clinical trials, reducing the bureaucratic burden on researchers, studying instances where the NIH missed out on identifying ground-breaking science, and finding new ways to get money into the hands of younger scientists.

The list reflects areas where the NIH has long acknowledged the need for improvement and has made efforts – sometimes extensive and prolonged – to get better.

Yet some of the demands also carry implicit criticism of the NIH and the academic research community, at a time when attempts at genuine improvement can also bring fears of politically motivated attempts to weaken public trust in science.

In particular, the language approved by Congress demanding the NIH undertake a “major independent study” of “near misses” alludes in unmistakable ways to the case of Katalin Karikó, who shared the 2023 Nobel Prize in Physiology or Medicine for her development of the mRNA technology used in Covid vaccines, despite suffering the near derailment of her career by the University of Pennsylvania’s lack of faith in her work.

The requirement by Congress to more systemically study the replication of science also involves persistent criticisms of the research process. But, said Stuart Buck, the executive director of the Good Science Project, a non-profit advocate of improvements in the nation’s system of science funding, the pursuit of reproducibility fundamentally should be seen as promoting an essential and basic part of quality control at any large operation.

“It should just be a routine thing” for a place such as the NIH, Dr Buck said.

In ordering the reproducibility study, lawmakers cited the scandal that came to light in 2022, when Science magazine reported evidence of intentional manipulation in data in a 2006 study of Alzheimer’s disease that fuelled years of studies and clinical trials based on the idea that Alzheimer’s arises from the formation of sticky plaques involving the protein known as beta-amyloid.

But research shortcomings as a whole, or just inconsistencies in data, could have a wide range of causes that needed to be regularly studied and understood, Dr Buck said. He cited the puzzle, from a decade ago, in which researchers studying breast cancer cells at both the Lawrence Berkeley National Laboratory and the Harvard University-affiliated Dana-Farber Cancer Institute could not understand why they kept having different results for the same tests.

After meticulous replication efforts involving cross-country in-person lab visits, they figured out that it was due to the different rates at which they stirred a particular enzyme in their work.

That case showed that even responsible top-tier researchers could generate different outcomes if reproducibility wasn’t carefully built into their work as a normal part of the process, Dr Buck said. “If replications come to different results, it doesn’t mean that the original results are fraudulent or even problematic,” he said.

Despite a decade of intensive study of reproducibility, there remained much to learn, said Brian Nosek, a professor of psychology at the University of Virginia, who is also co-founder and executive director of the Center for Open Science, which promotes improved research practices.

The questions still needing better answers included the essential practices for achieving reproducibility and replicability, and the conditions where the costs of improving replicability exceeded the benefits, Professor Nosek said.

There was, however, no question that the pursuit – broadly known as metascience – was worth substantial ongoing investment, Professor Nosek said. Arguing otherwise “is a bit like that old aphorism about the fellow who insists that he can’t stop to sharpen the saw because he is too busy sawing”, he said.

NIH officials said they planned to comply fully with the requirements set out by Congress, but did not yet have any specific comments on the matter. “We are in the process of reviewing these different requirements as we work to implement the new directives from Congress,” NIH officials said. “Our goal is always to be responsive to Congress and meet any new requirements in a timely fashion.”

In terms of funding, meanwhile, Congress largely flat-funded the NIH, keeping its annual budget at around $49 billion, in a year when the lawmakers cut the spending levels at other leading federal science agencies.

paul.basken@timeshighereducation.com