Post by marchesarosa on Oct 18, 2012 11:04:45 GMT 1
From Steve McIntyre on Climate Audit
climateaudit.org/2012/10/16/forensic-bioinformatics/#more-17077
Forensic Bioinformatics
Pielke Jr has sent me the following two links on the longstanding dispute between Baggerly and Coombes, two biostatisticians, against a team of cancer researchers at Duke University, led by young star Dr Potti. See CBS News here www.cbsnews.com/video/watch/?id=7398476n&tag=contentBody;storyMediaBox and a Baggerly 2010 lecture here videolectures.net/cancerbioinformatics2010_baggerly_irrh/
Baggerly and Coombes had attempted to replicate a leading paper; their efforts have ultimately led to retraction of the papers. But the decisive step in the retraction did not arise from proper operation of the peer review system or university investigations, but through something entirely fortuitous.
Their experience has many parallels to Climate Audit versus the Mann hockey stick, even to some small details. (This is not to say that all details are parallel). For example, the Potti et al papers used “meta genes”, which Baggerly explained as being nothing but principal components.
Like us, Baggerly and Coombes were frustrated by incomplete and/or misleading documentation and the resulting need to resort to what they described as “forensic bioinformatics”- which is exactly equivalent to what has been described at CA as reverse engineering. Like us, they even encountered a one-row-off error (compare to MM03). They described an incident where the Potti authors appear to have reversed labels (resistant-sensitive) on a drug – though the labels were reversed and not Mannian upside-down.
They published critical comments in journals. Potti and the original authors used Mannian language to rebuff the critiques, which they described as “deeply flawed”. The Potti coauthors said that the Baggerly and Coombes criticisms didn’t matter, that they were little more than complaints about typographical errors and that any given criticism was obsolete since subsequent studies had confirmed the results being criticized at any given time.
Baggerly and Coombes had trouble publishing results in cancer journals because their results were “too negative”. Eventually they published in Annals of Applied Statistics, leading to an investigation at Duke University.
But after three months, the Duke investigation completely cleared the Potti authors, stating (in language reminiscent of Muir Russell) that the investigation had “strengthened their confidence”. Baggerly and Coombes’ request for a copy of the investigation report was refused.
Eventually they noticed that a copy had been sent to the National Cancer Institute, a federally funded agency, and, in a tactic reminiscent of Climate Audit, they submitted a FOI request for the investigation report, receiving it in a redacted form. Needless to say, the investigation had failed to actually investigate the allegations.
Baggerly and Coombes were incredulous, but had more or less run out of avenues to pursue.
Then completely out of left field came a decisive revelation from “The Cancer Letter”, a weekly newsletter that is sort of like a blog (see here). In his CV, Dr Potti had falsely claimed to be a “Rhodes Scholar (Australia)” – a claim refuted at The Cancer Letter.
Although this misrepresentation did not bear on the dispute itself, it was the sort of thing that the academic community could dig its teeth into and Potti’s data manipulation began to unravel. Dr Nevens, Potti’s senior and coauthor at Duke, finally withdrew his support. The articles (published at the most eminent journals) were subsequently retracted.
There are many other parallels, but not everything is parallel. However, failing to report adverse results (e.g. a verification r2 of ~0 or hide the decline) are forms of data manipulation that should be taken seriously in the climate community, but aren’t.
climateaudit.org/2012/10/16/forensic-bioinformatics/#more-17077
Forensic Bioinformatics
Pielke Jr has sent me the following two links on the longstanding dispute between Baggerly and Coombes, two biostatisticians, against a team of cancer researchers at Duke University, led by young star Dr Potti. See CBS News here www.cbsnews.com/video/watch/?id=7398476n&tag=contentBody;storyMediaBox and a Baggerly 2010 lecture here videolectures.net/cancerbioinformatics2010_baggerly_irrh/
Baggerly and Coombes had attempted to replicate a leading paper; their efforts have ultimately led to retraction of the papers. But the decisive step in the retraction did not arise from proper operation of the peer review system or university investigations, but through something entirely fortuitous.
Their experience has many parallels to Climate Audit versus the Mann hockey stick, even to some small details. (This is not to say that all details are parallel). For example, the Potti et al papers used “meta genes”, which Baggerly explained as being nothing but principal components.
Like us, Baggerly and Coombes were frustrated by incomplete and/or misleading documentation and the resulting need to resort to what they described as “forensic bioinformatics”- which is exactly equivalent to what has been described at CA as reverse engineering. Like us, they even encountered a one-row-off error (compare to MM03). They described an incident where the Potti authors appear to have reversed labels (resistant-sensitive) on a drug – though the labels were reversed and not Mannian upside-down.
They published critical comments in journals. Potti and the original authors used Mannian language to rebuff the critiques, which they described as “deeply flawed”. The Potti coauthors said that the Baggerly and Coombes criticisms didn’t matter, that they were little more than complaints about typographical errors and that any given criticism was obsolete since subsequent studies had confirmed the results being criticized at any given time.
Baggerly and Coombes had trouble publishing results in cancer journals because their results were “too negative”. Eventually they published in Annals of Applied Statistics, leading to an investigation at Duke University.
But after three months, the Duke investigation completely cleared the Potti authors, stating (in language reminiscent of Muir Russell) that the investigation had “strengthened their confidence”. Baggerly and Coombes’ request for a copy of the investigation report was refused.
Eventually they noticed that a copy had been sent to the National Cancer Institute, a federally funded agency, and, in a tactic reminiscent of Climate Audit, they submitted a FOI request for the investigation report, receiving it in a redacted form. Needless to say, the investigation had failed to actually investigate the allegations.
Baggerly and Coombes were incredulous, but had more or less run out of avenues to pursue.
Then completely out of left field came a decisive revelation from “The Cancer Letter”, a weekly newsletter that is sort of like a blog (see here). In his CV, Dr Potti had falsely claimed to be a “Rhodes Scholar (Australia)” – a claim refuted at The Cancer Letter.
Although this misrepresentation did not bear on the dispute itself, it was the sort of thing that the academic community could dig its teeth into and Potti’s data manipulation began to unravel. Dr Nevens, Potti’s senior and coauthor at Duke, finally withdrew his support. The articles (published at the most eminent journals) were subsequently retracted.
There are many other parallels, but not everything is parallel. However, failing to report adverse results (e.g. a verification r2 of ~0 or hide the decline) are forms of data manipulation that should be taken seriously in the climate community, but aren’t.