You might have seen or heard people engaged in debates about evolution, climate change, GMOs, vaccinations, or any other popular science-oriented issue refer to peer- reviewed literature, published papers, or journal articles. These all refer to the same thing. So what is the big deal about peer-reviewed science literature, and why do some people keep harping on this as if it were the main standard for truth and integrity? Isn’t a scientific paper in a journal the same thing as a magazine article or an op-ed piece in the newspaper, or even a post on Facebook?
No, it’s not. Papers are the currency and lifeblood of science. They are what count. Speeches, lectures, presentations at prestigious meetings, articles in magazines, interviews on TV, blog posts and books are nice, but a scientist is judged by his or her publication record, first and foremost
It takes anywhere from three weeks to three months to write a paper. A good scientific paper can be as short as a page (like Watson and Crick’s announcement of the double helix structure of DNA), or as long as most of an issue of a journal (like the announcements of the human genome sequence). The number of authors can be one (pretty rare these days) to in the hundreds. Most papers (like most books) are not very important and are read by only a small number of people, usually colleagues in the same field as the topic of the paper. A small minority are read by hundreds. Almost no one who isn’t a scientist ever reads a scientific paper.
All papers share certain features. They are written in the passive voice. I use the passive voice a lot (“they are written in the passive voice”, being an example), which has gotten me in trouble when I write other stuff. The style is very particular, and not found in other kinds of writing. One way to describe it is extreme low key. One never uses phrases like “we have discovered”, or “we now prove that”, but instead things like “These results are consistent with the concept that humans and chimps evolved from a common ancestor”, which is about as strong a statement as you are likely to see.
What makes a scientific paper special is the fact that in order to get published it must be peer reviewed. When the paper is finished, the author sends it to a journal. The editor assigns the manuscript to two or more professional scientists who are working in the same field as the topic of the paper. These reviewers then read the manuscript and decide if the paper is good enough to be published. Papers can be rejected (fairly common for the top rated journals), accepted as is (very rare) or sent back to the author with suggestions for changes or questions that need to be answered. The comments (all of which must be addressed) can be minor or can require months of more experimental work.
After the author resubmits the revised manuscript, the whole process repeats, with the same reviewers judging the new version of the paper. From the time the paper is first submitted until the final accepted version is published, anywhere from a few months to over a year can go by. Reviewers generally do not get paid for their time or efforts in reviewing papers. It’s just considered to be part of the job of being a scientist.
When you read a paper in Science, Nature, the Proc. of the National Acad. Sciences, or any other journal, you know that that paper has been carefully and thoroughly checked by other scientists and has passed a stringent review. Of course the process is not perfect. Some reviewers miss things, and sometimes a bad paper gets through. But it usually works well, and the entire system of science relies on the proper functioning of the peer review approach.
There have been suggestions that scientists publish their work online, without peer review. This is a very bad idea. It’s also a very bad idea to trust anything scientific you might read on a website or a blog post (this one of course being an exception) unless there are citations to published peer reviewed papers included. Statements like “Many scientists have found that GMOs cause negative effects on human health” are meaningless. Statements like “Vaccination has been proven safe in numerous studies (References 1-10)” can be taken seriously, as long as one checks the references to see if the poster is telling the truth. (I have seen many statements where the referenced papers said the opposite of what was reported in the post.)
The peer review system is the best way we know to ensure honesty, quality, integrity and accuracy in reports of scientific findings. Without it, we would revert to the days when those who yelled the loudest or seemed the most sincere would prevail in the battlegrounds of ideas. The web can, at its worst, encourage that backward direction, and all the more reason for people to be careful and rely on peer-reviewed papers for truth in science.
The Book of Works 
Prior to surgery a few years ago, the surgeon handed me a marker pen and asked me to mark the foot that he was scheduled to work on. I am not a doctor but I was the person most qualified to identify which foot needed surgery.
The analogy carries into science also… Peer review is an important tool to check scientist’s work, but in many cases scientists are not the most qualified people to check the quality of their work. Frequently, the science requires mathematics and statistics beyond the capabilities of both the authors their peers. This is something that peer review does a poor job of addressing.
Having two doctors check the statistics of a third is like asking two carpenters to check the plumbing work of another carpenter.
We had a couple of major blow ups over the last couple of decades in climate science that has spoken loudly to this issue. Specifically: Karoly and Gergis’s paper in the Journal of Climate, 2012. A press release of the paper was quoted widely in newspapers all over the world prior to publication yet the paper was found to be deeply flawed by non-scientists within an hour of publication in the journal.
What Karoly, Gergis did was make an error and anyone can make an error, but their error was comparable to cutting off the wrong foot, it approached malpractice – and their peers did not catch it.
Some say it is because they used “pal review” instead of “peer review”. This is a common problem when the field is highly specialized and everyone knows each other’s work.
I absolutely agree that all scientific work must be peer-reviewed... but that is not nearly enough. To be science, one must be able to replicate a study – therefor all data, all algorithms, all information necessary to reproduce the study must be available to other scientists and if the public has paid for the science, it should be available to the public as well.
I understand that many scientists are reluctant to do this because they fear that advocates for positions they believe are wrong may misuse their data – but that is the cost of science. In the end, the facts will win.
Greg
I agree that “all the information necessary to reproduce a study must be available to other scientists…and the public”. The choice of who should review is not always easy. Editors can make mistakes and assign a manuscript to an enemy of the author, or to too close a friend. (I was once asked to review a paper by my ex wife. I declined). The real point of my post was to point out that without ANY peer review, anything can be published and pass as factual, as often happens on the net. There is no problem with that for many areas like politics, religion, social life (not to mention humor). But if science is done that way, the result is chaos and complete mayhem.
I agree completely.
My point was that research is getting so complex that scientists must often deal with multiple domains that are outside their area of expertise. Editors get tripped up by selecting peers from the author’s area rather than the domains critical to the subject matter.
Usually this is not a problem – but when the topic becomes critical and exposed to public scrutiny, the real experts pop out of the woodwork – and many of them are not scientist but practitioners in the subject matter area.
My favorite story along these lines was not about science – but the engineering of the Intel Pentium chip. in 1994, a division error in the chip was discovered by Dr. Thomas R. Nicely, a professor of mathematics at Lynchburg College. Dr. Nicely knew nothing about designing chips – but I believe that like a true professor, he said something to the effect of “Obviously someone did not check their work.”
Your point is a crucially important one, but I don’t think it goes far enough. Peer review does not involve other experts checking the results of others by revisiting their work in detail. Peer review is an evaluation of whether an attempt was made to use and follow appropriate methods. Publishing a peer-reviewed work does not mean that the results or conclusions drawn are correct, just that they deserve consideration by experts in the field. And so what laypeople should be looking at is not what this or that individual peer-reviewed article says, but what the consensus of experts in the field is.
Very well put, and of course I agree entirely. I have published at least 2 papers (that I know of) that turned out to be totally wrong, but that survived peer review, since there was no way any reviewer would know they were wrong. Thanks for the comment.