Addition Oct 2014: added link to the Reproducibility Initiative, now at validation.ScienceExchange.com

Correction Monday morning: the project is called the Reproducibility Initiative, not Project. Also, note that we got a comment from co-founder Elizabeth Iorns – discuss!

I happened to catch friend Ivan Oransky of RetractionWatch on NPR’s Science Friday last week, and caught an item that excites me because it could strengthen one of the pillars of medicine: scientific evidence. Would you believe that most published research is never re-tested by an independent lab?? Ivan talked about a new initiative to fix that, the Reproducibility Initiative, reported in August on Slate:

Precious research money is wasted on unreal results, but we can change the culture of science.

I’m a little embarrassed that we didn’t hear about this project until now, because we’re repeatedly covered this issue, and it’s never been good news. (I’ll paste in a list at bottom.)  In short, consider that your doctors – or you, if you’re a clinician – are taught to stick to the evidence. Yet:

• December 2010, New Yorker: “The Truth Wears Off“: If you took science classes in high school or younger it’ll gross you out, as it did me: we were taught that when another person repeats your experiment they dang well better get the same result, but most medical research is never tested again by a different researcher.
• A TED talk and TEDMED talk by Ben Goldacre MD explain that even when some studies are done by multiple labs, often some of the data goes missing.

(The list at bottom links to those posts.)

The opening issue of our Journal of Participatory Medicine had several pieces on weak evidence, some by Peter Frishauf, founder of Medscape and author of numerous JoPM articles (and one of the posts below).

Why aren’t studies replicated by independent labs?

I can’t say. I can only say it’s a farce that we trust unverified evidence. And this isn’t just a disservice to patients – it’s a disservice to the clinicians who are trained to trust peer-reviewed evidence.

Unverified evidence is no proof of fraud. The point is just that until someone else verifies it, we don’t know. And generally, science doesn’t verify. From the Slate piece:

NASA made a headline-grabbing announcement in 2010 that scientists had found bacteria that could live on arsenic—a finding that would require biology textbooks to be rewritten. At the time, many experts condemned the paper as a poor piece of science that shouldn’t have been published. This July, two teams of scientists reported that they couldn’t replicate the results. …

… C. Glenn Begley, who spent a decade in charge of global cancer research at the biotech giant Amgen, recently dispatched 100 Amgen scientists to replicate 53 landmark experiments in cancer—the kind of experiments that lead pharmaceutical companies to sink millions of dollars to turn the results into a drug. In March Begley published the results: They failed to replicate 47 of them.

Can you imagine that? 53 major cancer studies, and 47 of them can’t be reproduced?

There’s more – but let’s get to the new project.

 Here’s how it is supposed to work. Let’s say you have found a drug that shrinks tumors. You write up your results, which are sexy enough to get into Nature or some other big-name journal. You also send the Reproducibility Initiative the details of your experiment and request that someone reproduce it. A board of advisers matches you up with a company with the experience and technology to do the job. You pay them to do the job—Iorns estimates the bill for replication will be about 10 percent of the original research costs—and they report back whether they got the same results.

I just love this. In fact, if I were funding research, I’d tack on another 10% to have it verified – wouldn’t you? (Heck, when I worked in printing, proofreading and press checks were built into the price. Who would pay for an unproofread print job?)

Fittingly, one of the Project’s advisors is John Ioannidis, of “decline effect” fame. Hurrah!

Well, not everyone is impressed.

As we discussed it tonight on Twitter, Ivan added:

Not everyone thinks Reproducibility Initiative is a great idea, here’s @kanderson http://t.co/CnSdogsO

In it, Kent Anderson (a journal editor today, formerly of the New England Journal of Medicine) expresses skepticism about the sponsors’ motives and whether what they’re offering can solve the problem: “The Reproducibility Initiative — Solving a Problem, or Just Another Attempt to Draw on Research Funds?”

I can’t help but note our post last March in which two former editors of NEJM described the incredible difficulty of finding good content; a subsequent editor reversed one conflict-of-interest policy because there aren’t enough untainted authors.

Ivan also added this Ars Technica post:

Scientific reproducibility, for fun and profit
A new project will confirm scientists’ results are reproducible—if they pay.

Really, scientists?  I don’t know the authors of these pieces (I welcome discussion), but really? Considering the evidence that our research results are weak, is it really appropriate to react by questioning the motives behind a solution?

I’m open to discussion here, but first, the fundamentals: isn’t it important that our scientific process deliver reliable evidence as its first priority? Can any scientist object to verification of his/her supposed “findings”? Can we call it a finding before someone’s reproduced it?

Can we?

_____________

References – previous posts here on the subject:

• Feb 2009: MedPage: Negative Data on Seroquel Suppressed by Drug’s Maker
• August 2010: A Troubled Trifecta: Peer Review, Academia & Tenure
• January 2011: The Decline Effect: Is there something wrong with the scientific method? 
• October 2011: e-Patient Training via TED Talk: “Battling Bad Science”
• March 2012: Former NEJM editors on the corruption of American medicine
• April 2012: “The cancer at the core of evidence-based medicine”: Ben Goldacre on the missing data  (A strong summary of previous coverage, with impromptu interview with TEDMED speaker Ben Goldacre)