Stop The Presses: Retraction by the Numbers

Posted by socogal42 on June 18, 2015 in Uncategorized | Leave a comment

It’s been a rough few weeks for the journal Science…and science as a whole. The retraction of a well-covered study on the effect of conversations with LGBT canvassers on votes for same-sex marriages has attracted a lot of attention from social media and has led the New York Times to publish a list of recent high profile retractions under the headline “Retracted Scientific Studies: A Growing List”). Throw on top a scientist’s recent admission that he deliberately ran a poorly designed experiment in the hopes of creating a media storm (and succeeded) and you get articles declaring that “peer review is notoriously uncertain.” This is a disturbing conclusion considering peer review is the main (and, essentially, only) way of sharing results and providing a measurement of scientific success that allows scientists to get jobs, grants, and everything else they need to actually have a career.

Are we in a scientific decline? Are scientists really just making up their results in the hopes of publishing in Science or Nature? Do I need to compromise my ethics to get ahead in my scientific career? Let’s do the scientific thing and look at the data on retracted papers to figure out what’s really going on.

 

First: What We Talk About When We Talk About Peer Review

Getting a paper published in a well-respected scientific journal can be a long and drawn out process (it can literally take YEARS). Once you get your paper written (often in a format particular to one specific journal), your paper is passed on to an editor, who will either reject it outright or send it on the “peer review” part of the process. This involves the paper being sent to 2-3 scientists whose work is similar enough to yours that they can comment on the merits and problems of the paper. Often, you will be allowed to suggest certain people who you would like to serve as reviewers, along with a few who you would like excluded from the process. Based on their comments, your paper will then be accepted, rejected or recommended for resubmission after the reviewer’s comments (which can include asking for additional experiments) have been addressed.

The majority of journals- at least the ones that people have heard of- employ this peer review process. The idea is that by having scientific peers as reviewers who can understand the merits of your research (rather than an editor who may not be familiar with your particular field of study), the merits and problems with the study will be more accurately assessed. In a perfect world, this process would also weed out papers with falsified or fraudulent data- the kind of papers that are later retracted.

But we don’t live in a perfect world. Both the peer review process and the journals that employ them don’t operate without bias, and they amplify some of the problems within the scientific community (for more on this, see this post by our own Cailey Bromer). And so, papers get published that shouldn’t have- and sometimes, those papers have to be retracted.

How Many Retracted Papers Are There, Really?

Every article on a high-profile retracted paper will mention the fact that there has been a dramatic increase in the number of retracted papers over the last three decades. What do the data have to say about this, and what percentage of papers are being retracted?

There are two major studies looking at the incidence of retraction in the literature: Fang, Steen, and Casadevall (2012) looked at retracted biomedical science papers found through PubMed (the most popular database for scientific research articles), while Grieneisen and Zhang (2012), searched the STEM (science, technology, engineering and mathematics) literature through PubMed and other databases. The former examines 2,047 retracted papers published between 1973 and 2011; the latter looks at 4,211 papers from between 1980 and 2010. Both agree that the number of papers retracted has increased since 1980, with the increase in papers retracted for fraud starting in the 1990s and those retracted for other kinds of misconduct (such as plagiarism or double publication) starting to increase in 2005 (Fang 2012).

However, the percentage of papers retracted is extremely small; the journal with the highest number of individual retractions, Science, had 73 papers retracted out of 76,801 published, or 0.09% (Grieneisen 2012). So really, retraction is an extremely rare event.

 

Why Do Papers Get Retracted?

Initially it was thought that most papers were retracted due to scientific errors, as that was the most commonly cited reason in the journal’s retraction notices themselves (Nath 2006; Steen 2010). However, this appears not to be the case; studies have suggested that anywhere between 66-75% of retractions are due to fraud or misconduct (Fang 2012; Grieneisen 2012). This includes papers pulled for fabricated data, plagiarism, duplicate publication or other suspected research misconduct. 14% of retraction notices don’t even give a reason for the paper being pulled (Fang 2012). Even in papers retracted after the Office of Research Integrity found evidence of research misconduct, only 42.1% of the retraction notices mentioned that any misconduct had taken place (Resnik 2012).

This may sound more than a little terrifying, all but confirming the media assertion that peer review journals are full of papers based on made-up data. But upon further analysis, Grieneisen and colleagues found that 54% of papers retracted for research misconduct (a total of 391 papers) came from just 13 individual researchers, 9 of whom had 20 or more retractions each. The record-holder for fraud is Yoshitaka Fujii, an anesthesiologist from Japan, who had 183 of his papers retracted after statistical analysis revealed that the data he reported were mathematically impossible (read more about this case over at the fabulous Nautil.us). So while it may seem that an almost overwhelming number of scientific articles are being pulled for misconduct, there seem to be a few truly bad apples skewing the numbers.

 

Why The Increase in Retractions?

The data tell us the number of paper retractions- whether due to fraud, misconduct or error- but they don’t necessarily tell us why. By evaluating the amount of time between publication and retraction, Steen and colleagues were able to draw some conclusions as to why there has been such a steep rise in the number of retracted papers in recent years. For one, there has been a significant decrease in the amount of time between publishing an article and retracting it- from an average of 50 months prior to 2002 to 24 months afterwards. Another contribution to the increase is that journals are also more likely to reach farther back into their archives and reexamine older papers from authors who have been found guilty of research misconduct. Additionally, the creation of programs in recent years that can accurately detect plagiarism has also made it easier and quicker to retract duplicated or plagiarized papers  (Steen 2013).

The number of overall publications and the increased scrutiny of high profile research, two factors that are commonly cited as major reasons for the increase in retractions, were found to have little to no effect on the increased retraction rate. Between 1995 and 2005, the rate of retraction increased more than the rate of publication, indicating that the two are somewhat independent for that time period. And while fraudulent papers published in better known journals such as Science or Nature were more likely to be discovered, this factor had a negligible effect on the actual rate of retractions (Steen 2013).

 

What Does This Mean for Science?

To be quite honest: probably not that much.

The vast, vast majority of scientists are not fabricating their data, despite what the media may make it sound like. Yes, the peer review process is imperfect, and people like Yoshitaka Fujii are able to find ways to take advantage of these imperfections. But for every scientist like Fujii, there are hundreds, if not thousands, who are honest and ethical.

This doesn’t mean that there aren’t ways to improve the process to prevent bad science getting published. Adam Marcus and Dr. Ivan Oransky, the founders of Retraction Watch (a website dedicated to keeping track of retracted papers in the sciences) are probably some of the foremost experts on the effects and consequences of retractions. In a 2014 review paper in the Journal of Microbiology and Biology Education, Marcus and Oransky show that one of the biggest hurdles to understanding why retractions happen comes from the journal themselves. Retraction notices are not standard between journals and rarely provide any information as to the reason for the retraction. It is also extremely difficult to search for retracted papers (one of the reasons why the Retraction Watch site has been such a success), and many papers that have been officially retracted may not even have a retraction notice. As with much of science, more transparency seems like it might be the key to improving the way retracted papers are handled and reported (Marcus 2014).

So the next time you see an article with a headline like “Science, Now Under Scrutiny Itself”, remember the 99.91%. They should really be the ones to focus on.

 

Works Cited

Fang, F. C., Steen, R. G., & Casadevall, A. (2012). Misconduct accounts for the majority of retracted scientific publications. Proceedings of the National Academy of Sciences, 109(42), 17028-17033.

Grieneisen, M. L., & Zhang, M. (2012). A comprehensive survey of retracted articles from the scholarly literature.

Marcus, A., & Oransky, I. (2014). What Studies of Retractions Tell Us. Journal of microbiology & biology education, 15(2), 151.

Nath, S. B., Marcus, S. C., & Druss, B. G. (2006). Retractions in the research literature: misconduct or mistakes?. Medical Journal of Australia, 185(3), 152.

Resnik, D. B., & Dinse, G. E. (2012). Scientific retractions and corrections related to misconduct findings. Journal of medical ethics, medethics-2012.

Steen, R. G. (2010). Retractions in the scientific literature: is the incidence of research fraud increasing?. Journal of medical ethics, jme-2010.

Steen, R. G., Casadevall, A., & Fang, F. C. (2013). Why has the number of scientific retractions increased. PloS one, 8(7), e68397.