Jaw dropping: most cancer studies cannot be replicated
Read it and weep
During a decade as head of global cancer research at Amgen, C. Glenn Begley identified 53 “landmark” publications — papers in top journals, from reputable labs — for his team to reproduce. Begley sought to double-check the findings before trying to build on them for drug development.
Result: 47 of the 53 could not be replicated. He described his findings in a commentary piece published on Wednesday in the journal Nature.
… they and others fear the phenomenon is the product of a skewed system of incentives that has academics cutting corners to further their careers.
…The most common response by the challenged scientists was: “you didn’t do it right.” Indeed, cancer biology is fiendishly complex, noted Phil Sharp, a cancer biologist and Nobel laureate at the Massachusetts Institute of Technology.
Even in the most rigorous studies, the results might be reproducible only in very specific conditions, Sharp explained: “A cancer cell might respond one way in one set of conditions and another way in different conditions. I think a lot of the variability can come from that.”
…Part way through his project to reproduce promising studies, Begley met for breakfast at a cancer conference with the lead scientist of one of the problematic studies.
“We went through the paper line by line, figure by figure,” said Begley. “I explained that we re-did their experiment 50 times and never got their result. He said they’d done it six times and got this result once, but put it in the paper because it made the best story. It’s very disillusioning.”
… basic science studies are rarely “blinded” the way clinical trials are. That is, researchers know which cell line or mouse got a treatment or had cancer. That can be a problem when data are subject to interpretation, as a researcher who is intellectually invested in a theory is more likely to interpret ambiguous evidence in its favor.
Via Arnold Kling, who had one dry and one optimistic comment to make:
Good thing that research at universities is not corrupted by the profit motive, the way it is at drug companies.
My reaction to this story is somewhat optimistic. We can fix this problem. If government and other funders of research were to shift more resources toward replication, this would do two things. First, it would catch more bad science sooner. Second, it would take away some of the incentive to do bad science, because it would raise the risk of getting caught.
I’m less optimistic: the only reason these cancer studies were examined were because someone was hoping to cash in on them; I see no incentive in other areas, and none for governments (except maybe, to my surprise, the military). I’ll also add this: if something cannot be replicated, your results aren’t robust. That makes them unusable in the real world.
My take-away: incentive matter, whether in academia or private industry.