Nation of Beancounters

Jaw dropping: most cancer studies cannot be replicated

Posted in Culture of Firms, On The Contrary by Navin Kumar on April 2, 2012

Read it and weep

During a decade as head of global cancer research at Amgen, C. Glenn Begley identified 53 “landmark” publications — papers in top journals, from reputable labs — for his team to reproduce. Begley sought to double-check the findings before trying to build on them for drug development.

Result: 47 of the 53 could not be replicated. He described his findings in a commentary piece published on Wednesday in the journal Nature.

… they and others fear the phenomenon is the product of a skewed system of incentives that has academics cutting corners to further their careers.

…The most common response by the challenged scientists was: “you didn’t do it right.” Indeed, cancer biology is fiendishly complex, noted Phil Sharp, a cancer biologist and Nobel laureate at the Massachusetts Institute of Technology.

Even in the most rigorous studies, the results might be reproducible only in very specific conditions, Sharp explained: “A cancer cell might respond one way in one set of conditions and another way in different conditions. I think a lot of the variability can come from that.”

…Part way through his project to reproduce promising studies, Begley met for breakfast at a cancer conference with the lead scientist of one of the problematic studies.

“We went through the paper line by line, figure by figure,” said Begley. “I explained that we re-did their experiment 50 times and never got their result. He said they’d done it six times and got this result once, but put it in the paper because it made the best story. It’s very disillusioning.”

… basic science studies are rarely “blinded” the way clinical trials are. That is, researchers know which cell line or mouse got a treatment or had cancer. That can be a problem when data are subject to interpretation, as a researcher who is intellectually invested in a theory is more likely to interpret ambiguous evidence in its favor.

Via Arnold Kling, who had one dry and one optimistic comment to make:

Good thing that research at universities is not corrupted by the profit motive, the way it is at drug companies.

My reaction to this story is somewhat optimistic. We can fix this problem. If government and other funders of research were to shift more resources toward replication, this would do two things. First, it would catch more bad science sooner. Second, it would take away some of the incentive to do bad science, because it would raise the risk of getting caught.

I’m less optimistic: the only reason these cancer studies were examined were because someone was hoping to cash in on them; I see no incentive in other areas, and none for governments (except maybe, to my surprise, the military). I’ll also add this: if something cannot be replicated, your results aren’t robust. That makes them unusable in the real world.

My take-away: incentive matter, whether in academia or private industry.


2 Responses

Subscribe to comments with RSS.

  1. Arnav Kacker said, on April 2, 2012 at 3:02 pm

    I didn’t completely understand your comment at the end. Could you elaborate on that?

  2. Navin Kumar said, on April 3, 2012 at 8:03 pm

    In the sense that Kling means it.

    Usually, academics are seen as people in a Pure Quest for The Truth. But in order to decide who gets tenure, more pay, less teaching load, etc we create systems that incentivize research. Since it’s hard for university committees to evaluate research, they outsource (so to speak) such things to Journals. But as long as you create incentives to win, you create incentives to game the system.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: