why is peer review important in research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.25(3); 2014 Oct

Peer Review in Scientific Publications: Benefits, Critiques, & A Survival Guide

Jacalyn kelly.

1 Clinical Biochemistry, Department of Pediatric Laboratory Medicine, The Hospital for Sick Children, University of Toronto, Toronto, Ontario, Canada

Tara Sadeghieh

Khosrow adeli.

2 Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Canada

3 Chair, Communications and Publications Division (CPD), International Federation for Sick Clinical Chemistry (IFCC), Milan, Italy

The authors declare no conflicts of interest regarding publication of this article.

Peer review has been defined as a process of subjecting an author’s scholarly work, research or ideas to the scrutiny of others who are experts in the same field. It functions to encourage authors to meet the accepted high standards of their discipline and to control the dissemination of research data to ensure that unwarranted claims, unacceptable interpretations or personal views are not published without prior expert review. Despite its wide-spread use by most journals, the peer review process has also been widely criticised due to the slowness of the process to publish new findings and due to perceived bias by the editors and/or reviewers. Within the scientific community, peer review has become an essential component of the academic writing process. It helps ensure that papers published in scientific journals answer meaningful research questions and draw accurate conclusions based on professionally executed experimentation. Submission of low quality manuscripts has become increasingly prevalent, and peer review acts as a filter to prevent this work from reaching the scientific community. The major advantage of a peer review process is that peer-reviewed articles provide a trusted form of scientific communication. Since scientific knowledge is cumulative and builds on itself, this trust is particularly important. Despite the positive impacts of peer review, critics argue that the peer review process stifles innovation in experimentation, and acts as a poor screen against plagiarism. Despite its downfalls, there has not yet been a foolproof system developed to take the place of peer review, however, researchers have been looking into electronic means of improving the peer review process. Unfortunately, the recent explosion in online only/electronic journals has led to mass publication of a large number of scientific articles with little or no peer review. This poses significant risk to advances in scientific knowledge and its future potential. The current article summarizes the peer review process, highlights the pros and cons associated with different types of peer review, and describes new methods for improving peer review.

WHAT IS PEER REVIEW AND WHAT IS ITS PURPOSE?

Peer Review is defined as “a process of subjecting an author’s scholarly work, research or ideas to the scrutiny of others who are experts in the same field” ( 1 ). Peer review is intended to serve two primary purposes. Firstly, it acts as a filter to ensure that only high quality research is published, especially in reputable journals, by determining the validity, significance and originality of the study. Secondly, peer review is intended to improve the quality of manuscripts that are deemed suitable for publication. Peer reviewers provide suggestions to authors on how to improve the quality of their manuscripts, and also identify any errors that need correcting before publication.

HISTORY OF PEER REVIEW

The concept of peer review was developed long before the scholarly journal. In fact, the peer review process is thought to have been used as a method of evaluating written work since ancient Greece ( 2 ). The peer review process was first described by a physician named Ishaq bin Ali al-Rahwi of Syria, who lived from 854-931 CE, in his book Ethics of the Physician ( 2 ). There, he stated that physicians must take notes describing the state of their patients’ medical conditions upon each visit. Following treatment, the notes were scrutinized by a local medical council to determine whether the physician had met the required standards of medical care. If the medical council deemed that the appropriate standards were not met, the physician in question could receive a lawsuit from the maltreated patient ( 2 ).

The invention of the printing press in 1453 allowed written documents to be distributed to the general public ( 3 ). At this time, it became more important to regulate the quality of the written material that became publicly available, and editing by peers increased in prevalence. In 1620, Francis Bacon wrote the work Novum Organum, where he described what eventually became known as the first universal method for generating and assessing new science ( 3 ). His work was instrumental in shaping the Scientific Method ( 3 ). In 1665, the French Journal des sçavans and the English Philosophical Transactions of the Royal Society were the first scientific journals to systematically publish research results ( 4 ). Philosophical Transactions of the Royal Society is thought to be the first journal to formalize the peer review process in 1665 ( 5 ), however, it is important to note that peer review was initially introduced to help editors decide which manuscripts to publish in their journals, and at that time it did not serve to ensure the validity of the research ( 6 ). It did not take long for the peer review process to evolve, and shortly thereafter papers were distributed to reviewers with the intent of authenticating the integrity of the research study before publication. The Royal Society of Edinburgh adhered to the following peer review process, published in their Medical Essays and Observations in 1731: “Memoirs sent by correspondence are distributed according to the subject matter to those members who are most versed in these matters. The report of their identity is not known to the author.” ( 7 ). The Royal Society of London adopted this review procedure in 1752 and developed the “Committee on Papers” to review manuscripts before they were published in Philosophical Transactions ( 6 ).

Peer review in the systematized and institutionalized form has developed immensely since the Second World War, at least partly due to the large increase in scientific research during this period ( 7 ). It is now used not only to ensure that a scientific manuscript is experimentally and ethically sound, but also to determine which papers sufficiently meet the journal’s standards of quality and originality before publication. Peer review is now standard practice by most credible scientific journals, and is an essential part of determining the credibility and quality of work submitted.

IMPACT OF THE PEER REVIEW PROCESS

Peer review has become the foundation of the scholarly publication system because it effectively subjects an author’s work to the scrutiny of other experts in the field. Thus, it encourages authors to strive to produce high quality research that will advance the field. Peer review also supports and maintains integrity and authenticity in the advancement of science. A scientific hypothesis or statement is generally not accepted by the academic community unless it has been published in a peer-reviewed journal ( 8 ). The Institute for Scientific Information ( ISI ) only considers journals that are peer-reviewed as candidates to receive Impact Factors. Peer review is a well-established process which has been a formal part of scientific communication for over 300 years.

OVERVIEW OF THE PEER REVIEW PROCESS

The peer review process begins when a scientist completes a research study and writes a manuscript that describes the purpose, experimental design, results, and conclusions of the study. The scientist then submits this paper to a suitable journal that specializes in a relevant research field, a step referred to as pre-submission. The editors of the journal will review the paper to ensure that the subject matter is in line with that of the journal, and that it fits with the editorial platform. Very few papers pass this initial evaluation. If the journal editors feel the paper sufficiently meets these requirements and is written by a credible source, they will send the paper to accomplished researchers in the field for a formal peer review. Peer reviewers are also known as referees (this process is summarized in Figure 1 ). The role of the editor is to select the most appropriate manuscripts for the journal, and to implement and monitor the peer review process. Editors must ensure that peer reviews are conducted fairly, and in an effective and timely manner. They must also ensure that there are no conflicts of interest involved in the peer review process.

Overview of the review process

When a reviewer is provided with a paper, he or she reads it carefully and scrutinizes it to evaluate the validity of the science, the quality of the experimental design, and the appropriateness of the methods used. The reviewer also assesses the significance of the research, and judges whether the work will contribute to advancement in the field by evaluating the importance of the findings, and determining the originality of the research. Additionally, reviewers identify any scientific errors and references that are missing or incorrect. Peer reviewers give recommendations to the editor regarding whether the paper should be accepted, rejected, or improved before publication in the journal. The editor will mediate author-referee discussion in order to clarify the priority of certain referee requests, suggest areas that can be strengthened, and overrule reviewer recommendations that are beyond the study’s scope ( 9 ). If the paper is accepted, as per suggestion by the peer reviewer, the paper goes into the production stage, where it is tweaked and formatted by the editors, and finally published in the scientific journal. An overview of the review process is presented in Figure 1 .

WHO CONDUCTS REVIEWS?

Peer reviews are conducted by scientific experts with specialized knowledge on the content of the manuscript, as well as by scientists with a more general knowledge base. Peer reviewers can be anyone who has competence and expertise in the subject areas that the journal covers. Reviewers can range from young and up-and-coming researchers to old masters in the field. Often, the young reviewers are the most responsive and deliver the best quality reviews, though this is not always the case. On average, a reviewer will conduct approximately eight reviews per year, according to a study on peer review by the Publishing Research Consortium (PRC) ( 7 ). Journals will often have a pool of reviewers with diverse backgrounds to allow for many different perspectives. They will also keep a rather large reviewer bank, so that reviewers do not get burnt out, overwhelmed or time constrained from reviewing multiple articles simultaneously.

WHY DO REVIEWERS REVIEW?

Referees are typically not paid to conduct peer reviews and the process takes considerable effort, so the question is raised as to what incentive referees have to review at all. Some feel an academic duty to perform reviews, and are of the mentality that if their peers are expected to review their papers, then they should review the work of their peers as well. Reviewers may also have personal contacts with editors, and may want to assist as much as possible. Others review to keep up-to-date with the latest developments in their field, and reading new scientific papers is an effective way to do so. Some scientists use peer review as an opportunity to advance their own research as it stimulates new ideas and allows them to read about new experimental techniques. Other reviewers are keen on building associations with prestigious journals and editors and becoming part of their community, as sometimes reviewers who show dedication to the journal are later hired as editors. Some scientists see peer review as a chance to become aware of the latest research before their peers, and thus be first to develop new insights from the material. Finally, in terms of career development, peer reviewing can be desirable as it is often noted on one’s resume or CV. Many institutions consider a researcher’s involvement in peer review when assessing their performance for promotions ( 11 ). Peer reviewing can also be an effective way for a scientist to show their superiors that they are committed to their scientific field ( 5 ).

ARE REVIEWERS KEEN TO REVIEW?

A 2009 international survey of 4000 peer reviewers conducted by the charity Sense About Science at the British Science Festival at the University of Surrey, found that 90% of reviewers were keen to peer review ( 12 ). One third of respondents to the survey said they were happy to review up to five papers per year, and an additional one third of respondents were happy to review up to ten.

HOW LONG DOES IT TAKE TO REVIEW ONE PAPER?

On average, it takes approximately six hours to review one paper ( 12 ), however, this number may vary greatly depending on the content of the paper and the nature of the peer reviewer. One in every 100 participants in the “Sense About Science” survey claims to have taken more than 100 hours to review their last paper ( 12 ).

HOW TO DETERMINE IF A JOURNAL IS PEER REVIEWED

Ulrichsweb is a directory that provides information on over 300,000 periodicals, including information regarding which journals are peer reviewed ( 13 ). After logging into the system using an institutional login (eg. from the University of Toronto), search terms, journal titles or ISSN numbers can be entered into the search bar. The database provides the title, publisher, and country of origin of the journal, and indicates whether the journal is still actively publishing. The black book symbol (labelled ‘refereed’) reveals that the journal is peer reviewed.

THE EVALUATION CRITERIA FOR PEER REVIEW OF SCIENTIFIC PAPERS

As previously mentioned, when a reviewer receives a scientific manuscript, he/she will first determine if the subject matter is well suited for the content of the journal. The reviewer will then consider whether the research question is important and original, a process which may be aided by a literature scan of review articles.

Scientific papers submitted for peer review usually follow a specific structure that begins with the title, followed by the abstract, introduction, methodology, results, discussion, conclusions, and references. The title must be descriptive and include the concept and organism investigated, and potentially the variable manipulated and the systems used in the study. The peer reviewer evaluates if the title is descriptive enough, and ensures that it is clear and concise. A study by the National Association of Realtors (NAR) published by the Oxford University Press in 2006 indicated that the title of a manuscript plays a significant role in determining reader interest, as 72% of respondents said they could usually judge whether an article will be of interest to them based on the title and the author, while 13% of respondents claimed to always be able to do so ( 14 ).

The abstract is a summary of the paper, which briefly mentions the background or purpose, methods, key results, and major conclusions of the study. The peer reviewer assesses whether the abstract is sufficiently informative and if the content of the abstract is consistent with the rest of the paper. The NAR study indicated that 40% of respondents could determine whether an article would be of interest to them based on the abstract alone 60-80% of the time, while 32% could judge an article based on the abstract 80-100% of the time ( 14 ). This demonstrates that the abstract alone is often used to assess the value of an article.

The introduction of a scientific paper presents the research question in the context of what is already known about the topic, in order to identify why the question being studied is of interest to the scientific community, and what gap in knowledge the study aims to fill ( 15 ). The introduction identifies the study’s purpose and scope, briefly describes the general methods of investigation, and outlines the hypothesis and predictions ( 15 ). The peer reviewer determines whether the introduction provides sufficient background information on the research topic, and ensures that the research question and hypothesis are clearly identifiable.

The methods section describes the experimental procedures, and explains why each experiment was conducted. The methods section also includes the equipment and reagents used in the investigation. The methods section should be detailed enough that it can be used it to repeat the experiment ( 15 ). Methods are written in the past tense and in the active voice. The peer reviewer assesses whether the appropriate methods were used to answer the research question, and if they were written with sufficient detail. If information is missing from the methods section, it is the peer reviewer’s job to identify what details need to be added.

The results section is where the outcomes of the experiment and trends in the data are explained without judgement, bias or interpretation ( 15 ). This section can include statistical tests performed on the data, as well as figures and tables in addition to the text. The peer reviewer ensures that the results are described with sufficient detail, and determines their credibility. Reviewers also confirm that the text is consistent with the information presented in tables and figures, and that all figures and tables included are important and relevant ( 15 ). The peer reviewer will also make sure that table and figure captions are appropriate both contextually and in length, and that tables and figures present the data accurately.

The discussion section is where the data is analyzed. Here, the results are interpreted and related to past studies ( 15 ). The discussion describes the meaning and significance of the results in terms of the research question and hypothesis, and states whether the hypothesis was supported or rejected. This section may also provide possible explanations for unusual results and suggestions for future research ( 15 ). The discussion should end with a conclusions section that summarizes the major findings of the investigation. The peer reviewer determines whether the discussion is clear and focused, and whether the conclusions are an appropriate interpretation of the results. Reviewers also ensure that the discussion addresses the limitations of the study, any anomalies in the results, the relationship of the study to previous research, and the theoretical implications and practical applications of the study.

The references are found at the end of the paper, and list all of the information sources cited in the text to describe the background, methods, and/or interpret results. Depending on the citation method used, the references are listed in alphabetical order according to author last name, or numbered according to the order in which they appear in the paper. The peer reviewer ensures that references are used appropriately, cited accurately, formatted correctly, and that none are missing.

Finally, the peer reviewer determines whether the paper is clearly written and if the content seems logical. After thoroughly reading through the entire manuscript, they determine whether it meets the journal’s standards for publication,

and whether it falls within the top 25% of papers in its field ( 16 ) to determine priority for publication. An overview of what a peer reviewer looks for when evaluating a manuscript, in order of importance, is presented in Figure 2 .

How a peer review evaluates a manuscript

To increase the chance of success in the peer review process, the author must ensure that the paper fully complies with the journal guidelines before submission. The author must also be open to criticism and suggested revisions, and learn from mistakes made in previous submissions.

ADVANTAGES AND DISADVANTAGES OF THE DIFFERENT TYPES OF PEER REVIEW

The peer review process is generally conducted in one of three ways: open review, single-blind review, or double-blind review. In an open review, both the author of the paper and the peer reviewer know one another’s identity. Alternatively, in single-blind review, the reviewer’s identity is kept private, but the author’s identity is revealed to the reviewer. In double-blind review, the identities of both the reviewer and author are kept anonymous. Open peer review is advantageous in that it prevents the reviewer from leaving malicious comments, being careless, or procrastinating completion of the review ( 2 ). It encourages reviewers to be open and honest without being disrespectful. Open reviewing also discourages plagiarism amongst authors ( 2 ). On the other hand, open peer review can also prevent reviewers from being honest for fear of developing bad rapport with the author. The reviewer may withhold or tone down their criticisms in order to be polite ( 2 ). This is especially true when younger reviewers are given a more esteemed author’s work, in which case the reviewer may be hesitant to provide criticism for fear that it will damper their relationship with a superior ( 2 ). According to the Sense About Science survey, editors find that completely open reviewing decreases the number of people willing to participate, and leads to reviews of little value ( 12 ). In the aforementioned study by the PRC, only 23% of authors surveyed had experience with open peer review ( 7 ).

Single-blind peer review is by far the most common. In the PRC study, 85% of authors surveyed had experience with single-blind peer review ( 7 ). This method is advantageous as the reviewer is more likely to provide honest feedback when their identity is concealed ( 2 ). This allows the reviewer to make independent decisions without the influence of the author ( 2 ). The main disadvantage of reviewer anonymity, however, is that reviewers who receive manuscripts on subjects similar to their own research may be tempted to delay completing the review in order to publish their own data first ( 2 ).

Double-blind peer review is advantageous as it prevents the reviewer from being biased against the author based on their country of origin or previous work ( 2 ). This allows the paper to be judged based on the quality of the content, rather than the reputation of the author. The Sense About Science survey indicates that 76% of researchers think double-blind peer review is a good idea ( 12 ), and the PRC survey indicates that 45% of authors have had experience with double-blind peer review ( 7 ). The disadvantage of double-blind peer review is that, especially in niche areas of research, it can sometimes be easy for the reviewer to determine the identity of the author based on writing style, subject matter or self-citation, and thus, impart bias ( 2 ).

Masking the author’s identity from peer reviewers, as is the case in double-blind review, is generally thought to minimize bias and maintain review quality. A study by Justice et al. in 1998 investigated whether masking author identity affected the quality of the review ( 17 ). One hundred and eighteen manuscripts were randomized; 26 were peer reviewed as normal, and 92 were moved into the ‘intervention’ arm, where editor quality assessments were completed for 77 manuscripts and author quality assessments were completed for 40 manuscripts ( 17 ). There was no perceived difference in quality between the masked and unmasked reviews. Additionally, the masking itself was often unsuccessful, especially with well-known authors ( 17 ). However, a previous study conducted by McNutt et al. had different results ( 18 ). In this case, blinding was successful 73% of the time, and they found that when author identity was masked, the quality of review was slightly higher ( 18 ). Although Justice et al. argued that this difference was too small to be consequential, their study targeted only biomedical journals, and the results cannot be generalized to journals of a different subject matter ( 17 ). Additionally, there were problems masking the identities of well-known authors, introducing a flaw in the methods. Regardless, Justice et al. concluded that masking author identity from reviewers may not improve review quality ( 17 ).

In addition to open, single-blind and double-blind peer review, there are two experimental forms of peer review. In some cases, following publication, papers may be subjected to post-publication peer review. As many papers are now published online, the scientific community has the opportunity to comment on these papers, engage in online discussions and post a formal review. For example, online publishers PLOS and BioMed Central have enabled scientists to post comments on published papers if they are registered users of the site ( 10 ). Philica is another journal launched with this experimental form of peer review. Only 8% of authors surveyed in the PRC study had experience with post-publication review ( 7 ). Another experimental form of peer review called Dynamic Peer Review has also emerged. Dynamic peer review is conducted on websites such as Naboj, which allow scientists to conduct peer reviews on articles in the preprint media ( 19 ). The peer review is conducted on repositories and is a continuous process, which allows the public to see both the article and the reviews as the article is being developed ( 19 ). Dynamic peer review helps prevent plagiarism as the scientific community will already be familiar with the work before the peer reviewed version appears in print ( 19 ). Dynamic review also reduces the time lag between manuscript submission and publishing. An example of a preprint server is the ‘arXiv’ developed by Paul Ginsparg in 1991, which is used primarily by physicists ( 19 ). These alternative forms of peer review are still un-established and experimental. Traditional peer review is time-tested and still highly utilized. All methods of peer review have their advantages and deficiencies, and all are prone to error.

PEER REVIEW OF OPEN ACCESS JOURNALS

Open access (OA) journals are becoming increasingly popular as they allow the potential for widespread distribution of publications in a timely manner ( 20 ). Nevertheless, there can be issues regarding the peer review process of open access journals. In a study published in Science in 2013, John Bohannon submitted 304 slightly different versions of a fictional scientific paper (written by a fake author, working out of a non-existent institution) to a selected group of OA journals. This study was performed in order to determine whether papers submitted to OA journals are properly reviewed before publication in comparison to subscription-based journals. The journals in this study were selected from the Directory of Open Access Journals (DOAJ) and Biall’s List, a list of journals which are potentially predatory, and all required a fee for publishing ( 21 ). Of the 304 journals, 157 accepted a fake paper, suggesting that acceptance was based on financial interest rather than the quality of article itself, while 98 journals promptly rejected the fakes ( 21 ). Although this study highlights useful information on the problems associated with lower quality publishers that do not have an effective peer review system in place, the article also generalizes the study results to all OA journals, which can be detrimental to the general perception of OA journals. There were two limitations of the study that made it impossible to accurately determine the relationship between peer review and OA journals: 1) there was no control group (subscription-based journals), and 2) the fake papers were sent to a non-randomized selection of journals, resulting in bias.

JOURNAL ACCEPTANCE RATES

Based on a recent survey, the average acceptance rate for papers submitted to scientific journals is about 50% ( 7 ). Twenty percent of the submitted manuscripts that are not accepted are rejected prior to review, and 30% are rejected following review ( 7 ). Of the 50% accepted, 41% are accepted with the condition of revision, while only 9% are accepted without the request for revision ( 7 ).

SATISFACTION WITH THE PEER REVIEW SYSTEM

Based on a recent survey by the PRC, 64% of academics are satisfied with the current system of peer review, and only 12% claimed to be ‘dissatisfied’ ( 7 ). The large majority, 85%, agreed with the statement that ‘scientific communication is greatly helped by peer review’ ( 7 ). There was a similarly high level of support (83%) for the idea that peer review ‘provides control in scientific communication’ ( 7 ).

HOW TO PEER REVIEW EFFECTIVELY

The following are ten tips on how to be an effective peer reviewer as indicated by Brian Lucey, an expert on the subject ( 22 ):

1) Be professional

Peer review is a mutual responsibility among fellow scientists, and scientists are expected, as part of the academic community, to take part in peer review. If one is to expect others to review their work, they should commit to reviewing the work of others as well, and put effort into it.

2) Be pleasant

If the paper is of low quality, suggest that it be rejected, but do not leave ad hominem comments. There is no benefit to being ruthless.

3) Read the invite

When emailing a scientist to ask them to conduct a peer review, the majority of journals will provide a link to either accept or reject. Do not respond to the email, respond to the link.

4) Be helpful

Suggest how the authors can overcome the shortcomings in their paper. A review should guide the author on what is good and what needs work from the reviewer’s perspective.

5) Be scientific

The peer reviewer plays the role of a scientific peer, not an editor for proofreading or decision-making. Don’t fill a review with comments on editorial and typographic issues. Instead, focus on adding value with scientific knowledge and commenting on the credibility of the research conducted and conclusions drawn. If the paper has a lot of typographical errors, suggest that it be professionally proof edited as part of the review.

6) Be timely

Stick to the timeline given when conducting a peer review. Editors track who is reviewing what and when and will know if someone is late on completing a review. It is important to be timely both out of respect for the journal and the author, as well as to not develop a reputation of being late for review deadlines.

7) Be realistic

The peer reviewer must be realistic about the work presented, the changes they suggest and their role. Peer reviewers may set the bar too high for the paper they are editing by proposing changes that are too ambitious and editors must override them.

8) Be empathetic

Ensure that the review is scientific, helpful and courteous. Be sensitive and respectful with word choice and tone in a review.

Remember that both specialists and generalists can provide valuable insight when peer reviewing. Editors will try to get both specialised and general reviewers for any particular paper to allow for different perspectives. If someone is asked to review, the editor has determined they have a valid and useful role to play, even if the paper is not in their area of expertise.

10) Be organised

A review requires structure and logical flow. A reviewer should proofread their review before submitting it for structural, grammatical and spelling errors as well as for clarity. Most publishers provide short guides on structuring a peer review on their website. Begin with an overview of the proposed improvements; then provide feedback on the paper structure, the quality of data sources and methods of investigation used, the logical flow of argument, and the validity of conclusions drawn. Then provide feedback on style, voice and lexical concerns, with suggestions on how to improve.

In addition, the American Physiology Society (APS) recommends in its Peer Review 101 Handout that peer reviewers should put themselves in both the editor’s and author’s shoes to ensure that they provide what both the editor and the author need and expect ( 11 ). To please the editor, the reviewer should ensure that the peer review is completed on time, and that it provides clear explanations to back up recommendations. To be helpful to the author, the reviewer must ensure that their feedback is constructive. It is suggested that the reviewer take time to think about the paper; they should read it once, wait at least a day, and then re-read it before writing the review ( 11 ). The APS also suggests that Graduate students and researchers pay attention to how peer reviewers edit their work, as well as to what edits they find helpful, in order to learn how to peer review effectively ( 11 ). Additionally, it is suggested that Graduate students practice reviewing by editing their peers’ papers and asking a faculty member for feedback on their efforts. It is recommended that young scientists offer to peer review as often as possible in order to become skilled at the process ( 11 ). The majority of students, fellows and trainees do not get formal training in peer review, but rather learn by observing their mentors. According to the APS, one acquires experience through networking and referrals, and should therefore try to strengthen relationships with journal editors by offering to review manuscripts ( 11 ). The APS also suggests that experienced reviewers provide constructive feedback to students and junior colleagues on their peer review efforts, and encourages them to peer review to demonstrate the importance of this process in improving science ( 11 ).

The peer reviewer should only comment on areas of the manuscript that they are knowledgeable about ( 23 ). If there is any section of the manuscript they feel they are not qualified to review, they should mention this in their comments and not provide further feedback on that section. The peer reviewer is not permitted to share any part of the manuscript with a colleague (even if they may be more knowledgeable in the subject matter) without first obtaining permission from the editor ( 23 ). If a peer reviewer comes across something they are unsure of in the paper, they can consult the literature to try and gain insight. It is important for scientists to remember that if a paper can be improved by the expertise of one of their colleagues, the journal must be informed of the colleague’s help, and approval must be obtained for their colleague to read the protected document. Additionally, the colleague must be identified in the confidential comments to the editor, in order to ensure that he/she is appropriately credited for any contributions ( 23 ). It is the job of the reviewer to make sure that the colleague assisting is aware of the confidentiality of the peer review process ( 23 ). Once the review is complete, the manuscript must be destroyed and cannot be saved electronically by the reviewers ( 23 ).

COMMON ERRORS IN SCIENTIFIC PAPERS

When performing a peer review, there are some common scientific errors to look out for. Most of these errors are violations of logic and common sense: these may include contradicting statements, unwarranted conclusions, suggestion of causation when there is only support for correlation, inappropriate extrapolation, circular reasoning, or pursuit of a trivial question ( 24 ). It is also common for authors to suggest that two variables are different because the effects of one variable are statistically significant while the effects of the other variable are not, rather than directly comparing the two variables ( 24 ). Authors sometimes oversee a confounding variable and do not control for it, or forget to include important details on how their experiments were controlled or the physical state of the organisms studied ( 24 ). Another common fault is the author’s failure to define terms or use words with precision, as these practices can mislead readers ( 24 ). Jargon and/or misused terms can be a serious problem in papers. Inaccurate statements about specific citations are also a common occurrence ( 24 ). Additionally, many studies produce knowledge that can be applied to areas of science outside the scope of the original study, therefore it is better for reviewers to look at the novelty of the idea, conclusions, data, and methodology, rather than scrutinize whether or not the paper answered the specific question at hand ( 24 ). Although it is important to recognize these points, when performing a review it is generally better practice for the peer reviewer to not focus on a checklist of things that could be wrong, but rather carefully identify the problems specific to each paper and continuously ask themselves if anything is missing ( 24 ). An extremely detailed description of how to conduct peer review effectively is presented in the paper How I Review an Original Scientific Article written by Frederic G. Hoppin, Jr. It can be accessed through the American Physiological Society website under the Peer Review Resources section.

CRITICISM OF PEER REVIEW

A major criticism of peer review is that there is little evidence that the process actually works, that it is actually an effective screen for good quality scientific work, and that it actually improves the quality of scientific literature. As a 2002 study published in the Journal of the American Medical Association concluded, ‘Editorial peer review, although widely used, is largely untested and its effects are uncertain’ ( 25 ). Critics also argue that peer review is not effective at detecting errors. Highlighting this point, an experiment by Godlee et al. published in the British Medical Journal (BMJ) inserted eight deliberate errors into a paper that was nearly ready for publication, and then sent the paper to 420 potential reviewers ( 7 ). Of the 420 reviewers that received the paper, 221 (53%) responded, the average number of errors spotted by reviewers was two, no reviewer spotted more than five errors, and 35 reviewers (16%) did not spot any.

Another criticism of peer review is that the process is not conducted thoroughly by scientific conferences with the goal of obtaining large numbers of submitted papers. Such conferences often accept any paper sent in, regardless of its credibility or the prevalence of errors, because the more papers they accept, the more money they can make from author registration fees ( 26 ). This misconduct was exposed in 2014 by three MIT graduate students by the names of Jeremy Stribling, Dan Aguayo and Maxwell Krohn, who developed a simple computer program called SCIgen that generates nonsense papers and presents them as scientific papers ( 26 ). Subsequently, a nonsense SCIgen paper submitted to a conference was promptly accepted. Nature recently reported that French researcher Cyril Labbé discovered that sixteen SCIgen nonsense papers had been used by the German academic publisher Springer ( 26 ). Over 100 nonsense papers generated by SCIgen were published by the US Institute of Electrical and Electronic Engineers (IEEE) ( 26 ). Both organisations have been working to remove the papers. Labbé developed a program to detect SCIgen papers and has made it freely available to ensure publishers and conference organizers do not accept nonsense work in the future. It is available at this link: http://scigendetect.on.imag.fr/main.php ( 26 ).

Additionally, peer review is often criticized for being unable to accurately detect plagiarism. However, many believe that detecting plagiarism cannot practically be included as a component of peer review. As explained by Alice Tuff, development manager at Sense About Science, ‘The vast majority of authors and reviewers think peer review should detect plagiarism (81%) but only a minority (38%) think it is capable. The academic time involved in detecting plagiarism through peer review would cause the system to grind to a halt’ ( 27 ). Publishing house Elsevier began developing electronic plagiarism tools with the help of journal editors in 2009 to help improve this issue ( 27 ).

It has also been argued that peer review has lowered research quality by limiting creativity amongst researchers. Proponents of this view claim that peer review has repressed scientists from pursuing innovative research ideas and bold research questions that have the potential to make major advances and paradigm shifts in the field, as they believe that this work will likely be rejected by their peers upon review ( 28 ). Indeed, in some cases peer review may result in rejection of innovative research, as some studies may not seem particularly strong initially, yet may be capable of yielding very interesting and useful developments when examined under different circumstances, or in the light of new information ( 28 ). Scientists that do not believe in peer review argue that the process stifles the development of ingenious ideas, and thus the release of fresh knowledge and new developments into the scientific community.

Another issue that peer review is criticized for, is that there are a limited number of people that are competent to conduct peer review compared to the vast number of papers that need reviewing. An enormous number of papers published (1.3 million papers in 23,750 journals in 2006), but the number of competent peer reviewers available could not have reviewed them all ( 29 ). Thus, people who lack the required expertise to analyze the quality of a research paper are conducting reviews, and weak papers are being accepted as a result. It is now possible to publish any paper in an obscure journal that claims to be peer-reviewed, though the paper or journal itself could be substandard ( 29 ). On a similar note, the US National Library of Medicine indexes 39 journals that specialize in alternative medicine, and though they all identify themselves as “peer-reviewed”, they rarely publish any high quality research ( 29 ). This highlights the fact that peer review of more controversial or specialized work is typically performed by people who are interested and hold similar views or opinions as the author, which can cause bias in their review. For instance, a paper on homeopathy is likely to be reviewed by fellow practicing homeopaths, and thus is likely to be accepted as credible, though other scientists may find the paper to be nonsense ( 29 ). In some cases, papers are initially published, but their credibility is challenged at a later date and they are subsequently retracted. Retraction Watch is a website dedicated to revealing papers that have been retracted after publishing, potentially due to improper peer review ( 30 ).

Additionally, despite its many positive outcomes, peer review is also criticized for being a delay to the dissemination of new knowledge into the scientific community, and as an unpaid-activity that takes scientists’ time away from activities that they would otherwise prioritize, such as research and teaching, for which they are paid ( 31 ). As described by Eva Amsen, Outreach Director for F1000Research, peer review was originally developed as a means of helping editors choose which papers to publish when journals had to limit the number of papers they could print in one issue ( 32 ). However, nowadays most journals are available online, either exclusively or in addition to print, and many journals have very limited printing runs ( 32 ). Since there are no longer page limits to journals, any good work can and should be published. Consequently, being selective for the purpose of saving space in a journal is no longer a valid excuse that peer reviewers can use to reject a paper ( 32 ). However, some reviewers have used this excuse when they have personal ulterior motives, such as getting their own research published first.

RECENT INITIATIVES TOWARDS IMPROVING PEER REVIEW

F1000Research was launched in January 2013 by Faculty of 1000 as an open access journal that immediately publishes papers (after an initial check to ensure that the paper is in fact produced by a scientist and has not been plagiarised), and then conducts transparent post-publication peer review ( 32 ). F1000Research aims to prevent delays in new science reaching the academic community that are caused by prolonged publication times ( 32 ). It also aims to make peer reviewing more fair by eliminating any anonymity, which prevents reviewers from delaying the completion of a review so they can publish their own similar work first ( 32 ). F1000Research offers completely open peer review, where everything is published, including the name of the reviewers, their review reports, and the editorial decision letters ( 32 ).

PeerJ was founded by Jason Hoyt and Peter Binfield in June 2012 as an open access, peer reviewed scholarly journal for the Biological and Medical Sciences ( 33 ). PeerJ selects articles to publish based only on scientific and methodological soundness, not on subjective determinants of ‘impact ’, ‘novelty’ or ‘interest’ ( 34 ). It works on a “lifetime publishing plan” model which charges scientists for publishing plans that give them lifetime rights to publish with PeerJ, rather than charging them per publication ( 34 ). PeerJ also encourages open peer review, and authors are given the option to post the full peer review history of their submission with their published article ( 34 ). PeerJ also offers a pre-print review service called PeerJ Pre-prints, in which paper drafts are reviewed before being sent to PeerJ to publish ( 34 ).

Rubriq is an independent peer review service designed by Shashi Mudunuri and Keith Collier to improve the peer review system ( 35 ). Rubriq is intended to decrease redundancy in the peer review process so that the time lost in redundant reviewing can be put back into research ( 35 ). According to Keith Collier, over 15 million hours are lost each year to redundant peer review, as papers get rejected from one journal and are subsequently submitted to a less prestigious journal where they are reviewed again ( 35 ). Authors often have to submit their manuscript to multiple journals, and are often rejected multiple times before they find the right match. This process could take months or even years ( 35 ). Rubriq makes peer review portable in order to help authors choose the journal that is best suited for their manuscript from the beginning, thus reducing the time before their paper is published ( 35 ). Rubriq operates under an author-pay model, in which the author pays a fee and their manuscript undergoes double-blind peer review by three expert academic reviewers using a standardized scorecard ( 35 ). The majority of the author’s fee goes towards a reviewer honorarium ( 35 ). The papers are also screened for plagiarism using iThenticate ( 35 ). Once the manuscript has been reviewed by the three experts, the most appropriate journal for submission is determined based on the topic and quality of the paper ( 35 ). The paper is returned to the author in 1-2 weeks with the Rubriq Report ( 35 ). The author can then submit their paper to the suggested journal with the Rubriq Report attached. The Rubriq Report will give the journal editors a much stronger incentive to consider the paper as it shows that three experts have recommended the paper to them ( 35 ). Rubriq also has its benefits for reviewers; the Rubriq scorecard gives structure to the peer review process, and thus makes it consistent and efficient, which decreases time and stress for the reviewer. Reviewers also receive feedback on their reviews and most significantly, they are compensated for their time ( 35 ). Journals also benefit, as they receive pre-screened papers, reducing the number of papers sent to their own reviewers, which often end up rejected ( 35 ). This can reduce reviewer fatigue, and allow only higher-quality articles to be sent to their peer reviewers ( 35 ).

According to Eva Amsen, peer review and scientific publishing are moving in a new direction, in which all papers will be posted online, and a post-publication peer review will take place that is independent of specific journal criteria and solely focused on improving paper quality ( 32 ). Journals will then choose papers that they find relevant based on the peer reviews and publish those papers as a collection ( 32 ). In this process, peer review and individual journals are uncoupled ( 32 ). In Keith Collier’s opinion, post-publication peer review is likely to become more prevalent as a complement to pre-publication peer review, but not as a replacement ( 35 ). Post-publication peer review will not serve to identify errors and fraud but will provide an additional measurement of impact ( 35 ). Collier also believes that as journals and publishers consolidate into larger systems, there will be stronger potential for “cascading” and shared peer review ( 35 ).

CONCLUDING REMARKS

Peer review has become fundamental in assisting editors in selecting credible, high quality, novel and interesting research papers to publish in scientific journals and to ensure the correction of any errors or issues present in submitted papers. Though the peer review process still has some flaws and deficiencies, a more suitable screening method for scientific papers has not yet been proposed or developed. Researchers have begun and must continue to look for means of addressing the current issues with peer review to ensure that it is a full-proof system that ensures only quality research papers are released into the scientific community.

Advertisement

• Previous Issue
• Previous Article
• Next Article

The Peer Review Imperative

Threats to peer review, public trust in science and medicine, peer review matters: research quality and the public trust.

Michael M. Todd, M.D., served as Handling Editor for this article.

This article has a related Infographic on p. 17A.

Accepted for publication October 13, 2020.

• Split-Screen
• Article contents
• Figures & tables
• Supplementary Data
• Peer Review
• Open the PDF for in another window
• Cite Icon Cite
• Get Permissions
• Search Site

Evan D. Kharasch , Michael J. Avram , J. David Clark , Andrew J. Davidson , Timothy T. Houle , Jerrold H. Levy , Martin J. London , Daniel I. Sessler , Laszlo Vutskits; Peer Review Matters: Research Quality and the Public Trust. Anesthesiology 2021; 134:1–6 doi: https://doi.org/10.1097/ALN.0000000000003608

Download citation file:

• Ris (Zotero)
• Reference Manager

“Peer review grounds the public trust in the scientific and medical research enterprise…”

Image: Adobe Stock.

In an era of evidence-based medicine, peer review is an engine and protector of that evidence. Such evidence, vetted by and surviving the peer review process, serves to inform clinical decision-making, providing practitioners with the information to make diagnostic and therapeutic decisions. Unfortunately, there is recent and growing pressure to prioritize the speed of research dissemination, often at the expense of careful peer review. It is timely to remind readers and the public of the value brought by peer review, its benefits to patients, how much the public trust in science and medicine rests upon peer review, and how these have become vulnerable.

Peer review has been the foundation of scholarly publishing and scientific communication since the 1665 publication of the Philosophical Transactions of the Royal Society. The benefits and advantages of peer review in scientific research, and particularly medical research, are manifold and manifest. 1   Journals, editors, and peer reviewers hold serious responsibility as stewards of valid information, with accountability to the scientific community and an obligation to maintain the public trust. Anesthesiology states its aspiration and its responsibility on the cover of every issue: Trusted Evidence. Quality peer review (more specifically, closed or single-blind peer review, in which the identity of reviewers is confidential) is a foundational tenet of Anesthesiology.

Peer review grounds the public trust in the scientific and medical research enterprise, as well as the substantial public investment in scientific research. Peer review affords patients some degree of comfort in placing their trust in practitioners, knowing that they should be informed by the best possible, vetted evidence.

Quality peer review enriches and safeguards the scientific content, transparency, comprehensibility, and scientific integrity of published articles. It can enhance published research importance, originality, authenticity, scientific validity, adherence to experimental rigor, and correctness of results and interpretations and can identify errors in research execution. Peer review can help authors improve reporting quality, presentation clarity, and transparency, thereby enhancing comprehension and potential use by clinicians and scientists. Careful scrutiny can identify whether research has appropriate ethical principles, regulatory approvals, compliance, and equitable inclusion of both sexes. Peer review should consider the appropriateness of authorship and can detect duplicate publication, fabrication, falsification, plagiarism, and other misconduct.

Peer review should serve as a tempering factor on overenthusiastic authors and overstated conclusions, unwarranted extrapolations, conflation of association with causality, unsupported clinical recommendations, and spin. Spin is a well known, unfortunately common, and often insidious bias in the presentation and interpretation of results that seeks to convince readers that the beneficial effect of an experimental treatment exceeds what has actually been found or that minimizes untoward effects. 2–4  

Manuscripts often change substantially between the initial submission and the revised and improved published version. Improvement during the peer review process is not apparent to readers, who only see the final, published article, but is well known to authors, reviewers, and editors. Peer review is a defining difference in an era of proliferating predatory journals and other forms of research dissemination. Anesthesiology reviewers and editors devote considerable effort in service to helping authors improve their scientific communications, whether published in this journal or if ultimately elsewhere.

In the domain of clinical research, peer review does not change the scientific premise of an investigation, the hypothesis, or the study design, although it frequently improves their communication. Peer review does not change clinical research data, although it often corrects, enhances, or strengthens the statistical analysis of those data and can markedly improve their presentation and clarity. More importantly, peer review can assess, correct, and improve the interpretation, meaning, importance, and communication of research results—and importantly, confirm that conclusions emanate strictly from those results. Peer review may occasionally fundamentally revise or even reverse clinical research interpretations and recommendations. Each of these many functions enhances reader understanding and should ultimately improve patient care.

Peer review is not a guarantee of truth, and it can be imperfect. Medical history provides many examples of peer-reviewed research that was later found to be incorrect, typically through error or occasionally from misconduct. However, peer review certainly was and remains an essential initial check and quality control that has weeded out, or corrected before publication, innumerable reports of research of insufficient quality or veracity that otherwise would have been published and thereby become publicly accessible. Additionally, science should be “self-correcting,” and peer review is one of the most important factors responsible for such correction. Peer review remains an element by which medical science achieves the “self-correction” that drives progress.

Quality peer review does take time. So also do the initial preparation of manuscripts and the modifications made by authors in response to peer review. Anesthesiology endeavors to provide both quality and timely peer review. Our time to first decision averages only 16 days.

The increasing emphasis on fast research dissemination, often absent quality peer review, comes mostly but not exclusively because of the immediacy of the internet and broader media and societal trends. In an era in which the companies whose major product is the immediacy of information are the economic leaders (Facebook, Twitter, Google, and Apple), it is unsurprising that the immediacy of information is challenging that of quality as the value proposition in the research marketplace. Nevertheless, fast is not synonymous with good. We believe that sacrificing quality on the altar of speed is unwise, benefits no one (except perhaps authors), and may ultimately diminish trust in medical research and possibly even worsen clinical care.

Another recent societal problem is the growing spillover of political and media communication trends into scientific communication. Almost half of Americans believe that science researchers overstate the implications of their research, and three in four think “the biggest problem with news about scientific research findings is the way news reporters cover it.” 5   Scientific conclusions may be perverted through internet-based campaigns of disinformation and misinformation and dissemination of misleading and biased information. 6   This threatens the public trust in the scientific enterprise and scientific knowledge. 7   Social media has made science and health vulnerable to strategic manipulation. 7 , 8   It is also “leaving peer-reviewed communication behind as some scientists begin to worry less about their citation index (which takes years to develop) and more about their Twitter response (measurable in hours).” 8   Peer-reviewed journals cannot reverse these trends, but they can at least ensure that scientific conclusions when presented are correct and clearly stated.

In addition to the premium on dissemination speed versus peer review quality, a new variant of rapid clinical research dissemination has emerged that abrogates peer review entirely: preprints. Preprints are research reports that are posted by authors in a publicly accessible online repository in place of or before publication in a peer-reviewed scholarly journal. The preprint concept is decades old, rooted in physics and mathematics, in which authors traditionally sent their hand- or typewritten manuscript draft to a few colleagues for feedback before submitting it to a journal for publication. With the advent of the internet, this process was replaced by preprint servers and public posting. With the creation of a preprint server for biology and the life sciences (bioRxiv.org), the posting of unreviewed manuscripts by basic biomedical scientists has exploded in popularity and practice. Next came the creation of medRxiv.org, a publicly accessible preprint server for disseminating unpublished and unreviewed clinical research results in their “preliminary form” 9   and more so a call for research funders to require mandatory posting of their grantees’ research reports first on preprint servers before peer-reviewed publication. 10   Lack of peer review is the hallmark of preprints.

The main arguments offered by proponents of preprints are the free and near-immediate access to research results, claimed acceleration of the progress of research by immediate dissemination without peer review, and the assumption that articles will be improved by feedback from a wider group of readers alongside formal review by a few experts. Specifically claimed advantages of preprints are that they bypass the peer review process that adversely delays the dissemination of research results and “lifesaving cures” and “the months-long turnaround time of the publishing process and share findings with the community more quickly.” 11   In addition it is claimed that preprints address “researchers recently becoming vocally frustrated about the lengthy process of distributing research through the conventional pipelines, numerous laments decrying increasingly impractical demands of journals and reviewers, complicated dynamics at play from both authors and publishers that can affect time to press” and enable “sharing papers online before (or instead of) publication in peer-reviewed journals.” 11  

Preprints for clinical research have been justifiably criticized. 2 , 12–15   Most importantly, medical preprints lack safeguards afforded by peer review and increase the possibility of disseminating wrong or incorrectly interpreted results. Related concerns are that preprints are unnecessary for and potentially harmful to scientific progress and a significant threat with potential consequence to patient health and safety. Preprint server proponents “assume that most preprints would subsequently be peer reviewed,” 10   possibly before or after formal publication (if published), thus enabling correction or improvement (before or after publication). However, it is estimated that careful peer review of a manuscript takes 5 to 6 h. 1 , 16   It seems highly unlikely that busy scientists will surf the web in search of preprints on which to spend half a day providing concerted informative peer review.

Preprint enthusiasts claim that peer review after posting will provide scholarly input, facilitate preprint improvement, and enhance research quality. In fact, such peer review has been scant with biologic preprints, and it seems naïve to expect it with medical preprints. In reality, most preprints receive few comments, even fewer formal reviews, and many comments that are “counted” to support the notion that preprints do undergo peer review actually come through social media; a tweet is hardly a substantive review. The idea that comments on servers will replace quality peer review is not happening now and seems unlikely to transpire. Moreover, a survey found that the lack of peer review was an important reason why authors deliberately choose to post via preprint. 17   Additionally, postdissemination peer review takes longer than traditional prepublication peer review, and there remains concern by authors who do value peer review about the quality of the post-preprint peer review process and the quality of posted preprints. 17  

Preprint server proponents state “the work in question would be available to interested readers while these processes (peer review) take place, which is more or less what happens in physics today.” 10   The lives of patients are different than the lives of subatomic particles. Preprints deliberately “decouples the dissemination of manuscripts from the much slower process of evaluation and certification.” 10   However, it is exactly that coupling that validates clinical research, benefits patients, improves health, and engenders public trust.

The potential for free and unfettered distribution of raw, unvetted, and potentially incorrect information to be consumed by clinicians and patients cannot be called a medical advance. Use of such information by news outlets and online web services to promote “new” and “latest” research further misinforms the public and patients and is a disservice.

Relegating peer review to the realm of option and afterthought is not in the interest of research quality and integrity or of patients and public health. There is no apparent value in abrogating peer review of clinical research and all its many attendant benefits in ensuring the quality of clinical research available to practitioners and patients. Practitioners and patients have historically not seen the unreviewed manuscript submissions that eventually become revised peer-reviewed publications. Doing so now, given the sizable fraction of clinical research manuscripts that are rejected for publication and the substantial changes in most that are published, by providing the public with unreviewed preprints seems to carry considerable risk.

An additional problem is that the same research report can be posted on several preprint servers or websites or multiple versions may exist on the same preprint site. Various versions may be the same or different, and the final peer-reviewed published article (if it ever exists) may bear little semblance to the various posted versions, which remain freely available. Which version is correct? Availability of various differing reports of the same research risks competing or incorrect information and can only generate confusion. Scientific publishing decades ago banned publication of the same research in multiple journals owing to concerns about data integrity and inappropriate reuse. Restarting this now, via preprints, seems unwise—especially in medicine.

The public cannot and should not be expected to differentiate between posting and peer-reviewed publication. Unfortunately, and worse, even some practitioners do not understand the difference. Posting is often referred to erroneously as publication. Indeed, even the world’s most prestigious scientific journals refer to posting as publication. 18   Such conflation blurs the validity of information. That peer-reviewed publications and preprints both receive digital object identifiers further blurs their distinction and may give the latter more apparent credibility in the eyes of the lay public. The preprint community (servers and scientists) continues to claim simultaneously that preprints are and are not publications, depending on how such claims meet their proclivities. Although the bioRxiv server contains the disclaimer “readers should be aware that articles on bioRxiv have not been finalized by authors, might contain errors, and report information that has not yet been accepted or endorsed in any way by the scientific or medical community” on a web page, 19   it is not on the preprint itself for readers to see (perhaps this disclaimer, and the one below, should appear on the cover page of every preprint and as a footnote on every page). Fortunately, the medRxiv home page ( http://www.medrxiv.org ) states the following disclaimer: “Preprints are preliminary reports of work that have not been certified by peer review. They should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.” Then why bother?

The popularity of preprints in the basic science world has exploded in the last 5 yr, with the number of documents posted to preprint servers increasing exponentially. 20   While acknowledging the noble reasons given by preprint servers and authors for the dissemination of research by posting, three other apparent reasons are less noble. The first is competition for research funding. Major research funders ( e.g. , the National Institutes of Health) do not allow citation of unpublished manuscripts in grant applications but do allow citation of preprints. 21 , 22   The second is the preoccupation of authors with the speed of availability. There is a growing (and disappointing) trend of authors perceiving a need to claim priority (“we are the first to report…”), grounded perhaps on fear of being “scooped.” The third is the pursuit of academic promotion, which is based largely on the number of peer-reviewed publications listed on a curriculum vitae . We now see faculty listing preprints in the peer-reviewed research publications section of their curriculum vitae. All these drivers (priority, science advancement, reputational reward, and financial return) 7   are investigator centric. They are neither quality-centric nor patient-centric.

Who benefits if clinical research quality is sacrificed at the altar of speed? Certainly, it is not patients, public health, or the public trust in science, medicine, and the research enterprise. Enthusiasm for preprints seems to be emanating mostly from investigators, presumably because of academic or other incentives, 23   including the desire for prominence and further funding. Is this why we do medical research? Should we be investigator- or patient-centric?

Little in the argumentation espoused by proponents of clinical preprints attends to their benefit to patients. Indeed, posted preprints without all the scrutiny and benefits of peer review may lack quality and validity and may report flawed data and conclusions, which may hurt patients. 17 , 23   As stated previously, “clinical studies of poor quality can harm patients who might start or stop therapy in response to faulty data, whereas little short-term harm would be expected from an unreviewed astronomy study.” 12  

The importance of peer review in clinical research and the downside of its absence in posted preprints is illuminated by the COVID-19 pandemic. As of this date (October 1, 2020), there are 9,222 unreviewed COVID-19 SARS–CoV-2 preprints posted: 7,257 on medRxiv and 1,965 on bioRxiv. 24   To date, 33 COVID-19 articles have been retracted (0.37%), and 5 others have been temporarily retracted or have expressions of concern. 25   Of the 33 retractions, 11 (33%) were posted on an Rxiv server. The overall retraction rate in the general peer-reviewed literature is 0.04%. 26  

Based upon one of the unreviewed COVID-19 medical preprints, 27   the Commissioner of the U.S. Food and Drug Administration (the government agency entrusted more than any other to protect public health) and the President of the United States announced that convalescent plasma from COVID-19 survivors was “safe and very effective” and had been “proven to reduce mortality by 35%.” 28   Although the Commissioner later, after scientific uproar over that misinformation, “corrected” his comment in a tweet (a back page retraction to a front page headline), 29   the preprint was used to justify a Food and Drug Administration decision to issue an emergency use authorization for convalescent plasma to treat severe COVID-19. Would these errors have been prevented by peer review? We will never know.

Even if priority in clinical (and basic) research is valued, compared to the unquestionable value of quality, clinical preprints have questionable necessity in establishing precedence in contemporary times. Clinical trials registration, which makes fully public the existence of all such research, establishes both who is doing what and when. Some investigators may even publish their entire clinical protocol, to further make their studies known and by whom and when.

For hundreds of years, patent medicines (exotic concoctions of substances, often addicting and sometimes toxic) were claimed to prevent or cure a panoply of illnesses, without any evidence of effectiveness or safety or warning of potential harm. These medical elixirs, the magic potions of snake oil salesmen and charlatans, were heavily advertised and promoted to ailing, sometimes desperate, and thoroughly unsuspecting citizens—all without any oversight, regulation, quality control, or peer review. It was not until the 20th century that medical peer review and the requirement for evidence of effectiveness and safety reigned in the “Wild West” and launched the modern era of medicine, yielding the scientific discovery, progress, and improvement in human health seen today. This era rests on the bedrock of peer review, the quality ideal, and the evidence that constitutes the foundation for evidence-based medicine.

Will clinical preprints become the patent medicines of the new millennium? Do they portend the unrestricted and unregulated spillage of anything claimed as research, by anyone, and absent the quality control afforded by peer review? Like the patent medicines of a bygone era, which were heavily promoted by the newly developed advertising industry, will “posted” clinical research become fodder for the medical advertising industry and media at large, pushing who knows what information and claims on practitioners and a public already deluged with endless promotions and claims with which they cannot keep up or verify? An unsuspecting public is incapable of differentiating between the “posting” of any research observation by anyone with access to a computer and proper scholarly “publication” of peer-reviewed results and conclusions. This is particularly true of vulnerable patients with severe and/or incurable diseases, who may grasp at anything. Moreover, continuous claims of “breakthroughs” and “proven treatments” based on preprints, followed by backpedaling after challenges and outcries, further reduces public confidence in the scientific endeavor as a whole. This can create the perception that clinical science is unreliable and might be a matter of turf wars and politics instead of reliable valid evidence.

Over the past century and throughout the world, legislation has been passed and government agencies have been created to protect the public and maintain their trust in the medicines they take. Few would advocate dismantling the protections against patent medicines. Why now consider dismantling the peer review process in clinical research?

In 2019, the editors of several journals expressed a well articulated principle that they will not accept clinical research manuscripts that had been previously posted to a preprint server. 30   Their rationale was that the benefit of preprint servers in clinical research did not outweigh the potential harm to patients and scientific integrity. Major specific concerns included: “1) Preprints may be perceived by some (and used by less scrupulous investigators) as evidence even though the studies have not gone through peer review and the public may not be able to discern an unreviewed preprint from a seminal article in a leading journal; 2) It seems unlikely that the kind of prepublication dialogue that has taken place in other academic disciplines (e.g. mathematics and physics) will take place in medicine or surgery because the incentives are very different; 3) Preprints may lead to multiple competing, and perhaps even conflicting, versions of the ‘same’ content being available online at the same time, which can cause (at least) confusion and (at most) grave harm; and 4) For the vast majority of medical diagnoses, a few months of review of a study’s findings do not make a difference; the pace of discovery and dissemination generally is adequate.” These editors’ concerns and approach merit consideration if not more widespread adoption.

The potential for practitioner and public confusion regarding the difference between unregulated preprints and peer-reviewed publication is substantial. Indeed, the posting of preprints is often incorrectly termed “publication.” Peer-reviewed publications versus posted “publications” will soon become a difference without a distinction. Moreover, authors cannot have it both ways. They cannot claim a preprint as a publication for purposes of a grant (and now in some universities potentially for purposes of a degree, appointment, and/or promotion), yet claim it is not a publication for the purposes of submission to a peer-reviewed journal that does not allow prior publication. More importantly, the peer review imperative in clinical research and the role it plays in research quality, the evidence base, and patient care, constitutes an obligation to patient safety that cannot and should not be abrogated.

Peer review, clinical research quality, and the public trust in clinical research all now face an unprecedented assault. Quality peer review is a foundational tenet of Anesthesiology and underlies the Trusted Evidence we publish. Quality, timely, and unpressured peer review will continue to be a hallmark of Anesthesiology , in service to readers, patients, and the public trust.

Acknowledgments

We thank Ryan Walther, Managing Editor, and Vicki Tedeschi, Director of Digital Communications, for their valuable insights.

Competing Interests

Dr. Clark has a consulting agreement with Teikoku Pharma USA (San Jose, California). Dr. Levy reports being on Advisory and Steering Committees for Instrumentation Laboratory (Bedford, Massachusetts), Merck & Co. (Kenilworth, New Jersey), and Octapharma (Lachen, Switzerland). Dr. London reports financial relationships with Wolters Kluwer UptoDate (Philadelphia, Pennsylvania) and Springer (journal honorarium; New York, New York). The remaining authors declare no competing interests.

Citing articles via

Most viewed, email alerts, related articles, social media, affiliations.

• ASA Practice Parameters
• Online First
• Author Resource Center
• About the Journal
• Editorial Board
• Rights & Permissions
• Online ISSN 1528-1175
• Print ISSN 0003-3022
• Anesthesiology
• ASA Monitor

• Terms & Conditions Privacy Policy
• Manage Cookie Preferences
• © Copyright 2024 American Society of Anesthesiologists

This Feature Is Available To Subscribers Only

Sign In or Create an Account

• Science Notes Posts
• Contact Science Notes
• Todd Helmenstine Biography
• Anne Helmenstine Biography
• Free Printable Periodic Tables (PDF and PNG)
• Periodic Table Wallpapers
• Interactive Periodic Table
• Periodic Table Posters
• How to Grow Crystals
• Chemistry Projects
• Fire and Flames Projects
• Holiday Science
• Chemistry Problems With Answers
• Physics Problems
• Unit Conversion Example Problems
• Chemistry Worksheets
• Biology Worksheets
• Periodic Table Worksheets
• Physical Science Worksheets
• Science Lab Worksheets
• My Amazon Books

Understanding Peer Review in Science

Peer review is an essential element of the scientific publishing process that helps ensure that research articles are evaluated, critiqued, and improved before release into the academic community. Take a look at the significance of peer review in scientific publications, the typical steps of the process, and and how to approach peer review if you are asked to assess a manuscript.

What Is Peer Review?

Peer review is the evaluation of work by peers, who are people with comparable experience and competency. Peers assess each others’ work in educational settings, in professional settings, and in the publishing world. The goal of peer review is improving quality, defining and maintaining standards, and helping people learn from one another.

In the context of scientific publication, peer review helps editors determine which submissions merit publication and improves the quality of manuscripts prior to their final release.

Types of Peer Review for Manuscripts

There are three main types of peer review:

• Single-blind review: The reviewers know the identities of the authors, but the authors do not know the identities of the reviewers.
• Double-blind review: Both the authors and reviewers remain anonymous to each other.
• Open peer review: The identities of both the authors and reviewers are disclosed, promoting transparency and collaboration.

There are advantages and disadvantages of each method. Anonymous reviews reduce bias but reduce collaboration, while open reviews are more transparent, but increase bias.

Key Elements of Peer Review

Proper selection of a peer group improves the outcome of the process:

• Expertise : Reviewers should possess adequate knowledge and experience in the relevant field to provide constructive feedback.
• Objectivity : Reviewers assess the manuscript impartially and without personal bias.
• Confidentiality : The peer review process maintains confidentiality to protect intellectual property and encourage honest feedback.
• Timeliness : Reviewers provide feedback within a reasonable timeframe to ensure timely publication.

Steps of the Peer Review Process

The typical peer review process for scientific publications involves the following steps:

• Submission : Authors submit their manuscript to a journal that aligns with their research topic.
• Editorial assessment : The journal editor examines the manuscript and determines whether or not it is suitable for publication. If it is not, the manuscript is rejected.
• Peer review : If it is suitable, the editor sends the article to peer reviewers who are experts in the relevant field.
• Reviewer feedback : Reviewers provide feedback, critique, and suggestions for improvement.
• Revision and resubmission : Authors address the feedback and make necessary revisions before resubmitting the manuscript.
• Final decision : The editor makes a final decision on whether to accept or reject the manuscript based on the revised version and reviewer comments.
• Publication : If accepted, the manuscript undergoes copyediting and formatting before being published in the journal.

Pros and Cons

While the goal of peer review is improving the quality of published research, the process isn’t without its drawbacks.

• Quality assurance : Peer review helps ensure the quality and reliability of published research.
• Error detection : The process identifies errors and flaws that the authors may have overlooked.
• Credibility : The scientific community generally considers peer-reviewed articles to be more credible.
• Professional development : Reviewers can learn from the work of others and enhance their own knowledge and understanding.
• Time-consuming : The peer review process can be lengthy, delaying the publication of potentially valuable research.
• Bias : Personal biases of reviews impact their evaluation of the manuscript.
• Inconsistency : Different reviewers may provide conflicting feedback, making it challenging for authors to address all concerns.
• Limited effectiveness : Peer review does not always detect significant errors or misconduct.
• Poaching : Some reviewers take an idea from a submission and gain publication before the authors of the original research.

Steps for Conducting Peer Review of an Article

Generally, an editor provides guidance when you are asked to provide peer review of a manuscript. Here are typical steps of the process.

• Accept the right assignment: Accept invitations to review articles that align with your area of expertise to ensure you can provide well-informed feedback.
• Manage your time: Allocate sufficient time to thoroughly read and evaluate the manuscript, while adhering to the journal’s deadline for providing feedback.
• Read the manuscript multiple times: First, read the manuscript for an overall understanding of the research. Then, read it more closely to assess the details, methodology, results, and conclusions.
• Evaluate the structure and organization: Check if the manuscript follows the journal’s guidelines and is structured logically, with clear headings, subheadings, and a coherent flow of information.
• Assess the quality of the research: Evaluate the research question, study design, methodology, data collection, analysis, and interpretation. Consider whether the methods are appropriate, the results are valid, and the conclusions are supported by the data.
• Examine the originality and relevance: Determine if the research offers new insights, builds on existing knowledge, and is relevant to the field.
• Check for clarity and consistency: Review the manuscript for clarity of writing, consistent terminology, and proper formatting of figures, tables, and references.
• Identify ethical issues: Look for potential ethical concerns, such as plagiarism, data fabrication, or conflicts of interest.
• Provide constructive feedback: Offer specific, actionable, and objective suggestions for improvement, highlighting both the strengths and weaknesses of the manuscript. Don’t be mean.
• Organize your review: Structure your review with an overview of your evaluation, followed by detailed comments and suggestions organized by section (e.g., introduction, methods, results, discussion, and conclusion).
• Be professional and respectful: Maintain a respectful tone in your feedback, avoiding personal criticism or derogatory language.
• Proofread your review: Before submitting your review, proofread it for typos, grammar, and clarity.
• Couzin-Frankel J (September 2013). “Biomedical publishing. Secretive and subjective, peer review proves resistant to study”. Science . 341 (6152): 1331. doi: 10.1126/science.341.6152.1331
• Lee, Carole J.; Sugimoto, Cassidy R.; Zhang, Guo; Cronin, Blaise (2013). “Bias in peer review”. Journal of the American Society for Information Science and Technology. 64 (1): 2–17. doi: 10.1002/asi.22784
• Slavov, Nikolai (2015). “Making the most of peer review”. eLife . 4: e12708. doi: 10.7554/eLife.12708
• Spier, Ray (2002). “The history of the peer-review process”. Trends in Biotechnology . 20 (8): 357–8. doi: 10.1016/S0167-7799(02)01985-6
• Squazzoni, Flaminio; Brezis, Elise; Marušić, Ana (2017). “Scientometrics of peer review”. Scientometrics . 113 (1): 501–502. doi: 10.1007/s11192-017-2518-4

Related Posts

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 16 April 2024

Structure peer review to make it more robust

• Mario Malički 0

Mario Malički is associate director of the Stanford Program on Research Rigor and Reproducibility (SPORR) and co-editor-in-chief of the Research Integrity and Peer Review journal.

You can also search for this author in PubMed   Google Scholar

You have full access to this article via your institution.

In February, I received two peer-review reports for a manuscript I’d submitted to a journal. One report contained 3 comments, the other 11. Apart from one point, all the feedback was different. It focused on expanding the discussion and some methodological details — there were no remarks about the study’s objectives, analyses or limitations.

My co-authors and I duly replied, working under two assumptions that are common in scholarly publishing: first, that anything the reviewers didn’t comment on they had found acceptable for publication; second, that they had the expertise to assess all aspects of our manuscript. But, as history has shown, those assumptions are not always accurate (see Lancet 396 , 1056; 2020 ). And through the cracks, inaccurate, sloppy and falsified research can slip.

As co-editor-in-chief of the journal Research Integrity and Peer Review (an open-access journal published by BMC, which is part of Springer Nature), I’m invested in ensuring that the scholarly peer-review system is as trustworthy as possible. And I think that to be robust, peer review needs to be more structured. By that, I mean that journals should provide reviewers with a transparent set of questions to answer that focus on methodological, analytical and interpretative aspects of a paper.

For example, editors might ask peer reviewers to consider whether the methods are described in sufficient detail to allow another researcher to reproduce the work, whether extra statistical analyses are needed, and whether the authors’ interpretation of the results is supported by the data and the study methods. Should a reviewer find anything unsatisfactory, they should provide constructive criticism to the authors. And if reviewers lack the expertise to assess any part of the manuscript, they should be asked to declare this.

Anonymizing peer review makes the process more just

Other aspects of a study, such as novelty, potential impact, language and formatting, should be handled by editors, journal staff or even machines, reducing the workload for reviewers.

The list of questions reviewers will be asked should be published on the journal’s website, allowing authors to prepare their manuscripts with this process in mind. And, as others have argued before, review reports should be published in full. This would allow readers to judge for themselves how a paper was assessed, and would enable researchers to study peer-review practices.

To see how this works in practice, since 2022 I’ve been working with the publisher Elsevier on a pilot study of structured peer review in 23 of its journals, covering the health, life, physical and social sciences. The preliminary results indicate that, when guided by the same questions, reviewers made the same initial recommendation about whether to accept, revise or reject a paper 41% of the time, compared with 31% before these journals implemented structured peer review. Moreover, reviewers’ comments were in agreement about specific parts of a manuscript up to 72% of the time ( M. Malički and B. Mehmani Preprint at bioRxiv https://doi.org/mrdv; 2024 ). In my opinion, reaching such agreement is important for science, which proceeds mainly through consensus.

Stop the peer-review treadmill. I want to get off

I invite editors and publishers to follow in our footsteps and experiment with structured peer reviews. Anyone can trial our template questions (see go.nature.com/4ab2ppc ), or tailor them to suit specific fields or study types. For instance, mathematics journals might also ask whether referees agree with the logic or completeness of a proof. Some journals might ask reviewers if they have checked the raw data or the study code. Publications that employ editors who are less embedded in the research they handle than are academics might need to include questions about a paper’s novelty or impact.

Scientists can also use these questions, either as a checklist when writing papers or when they are reviewing for journals that don’t apply structured peer review.

Some journals — including Proceedings of the National Academy of Sciences , the PLOS family of journals, F1000 journals and some Springer Nature journals — already have their own sets of structured questions for peer reviewers. But, in general, these journals do not disclose the questions they ask, and do not make their questions consistent. This means that core peer-review checks are still not standardized, and reviewers are tasked with different questions when working for different journals.

Some might argue that, because different journals have different thresholds for publication, they should adhere to different standards of quality control. I disagree. Not every study is groundbreaking, but scientists should view quality control of the scientific literature in the same way as quality control in other sectors: as a way to ensure that a product is safe for use by the public. People should be able to see what types of check were done, and when, before an aeroplane was approved as safe for flying. We should apply the same rigour to scientific research.

Ultimately, I hope for a future in which all journals use the same core set of questions for specific study types and make all of their review reports public. I fear that a lack of standard practice in this area is delaying the progress of science.

Nature 628 , 476 (2024)

doi: https://doi.org/10.1038/d41586-024-01101-9

Reprints and permissions

Competing Interests

M.M. is co-editor-in-chief of the Research Integrity and Peer Review journal that publishes signed peer review reports alongside published articles. He is also the chair of the European Association of Science Editors Peer Review Committee.

Related Articles

• Scientific community
• Peer review

Londoners see what a scientist looks like up close in 50 photographs

Career News 18 APR 24

Researchers want a ‘nutrition label’ for academic-paper facts

Nature Index 17 APR 24

Deadly diseases and inflatable suits: how I found my niche in virology research

Spotlight 17 APR 24

Is ChatGPT corrupting peer review? Telltale words hint at AI use

News 10 APR 24

Three ways ChatGPT helps me in my academic writing

Career Column 08 APR 24

Is AI ready to mass-produce lay summaries of research articles?

Nature Index 20 MAR 24

Postdoctoral Position

We are seeking highly motivated and skilled candidates for postdoctoral fellow positions

Boston, Massachusetts (US)

Boston Children's Hospital (BCH)

Qiushi Chair Professor

Distinguished scholars with notable achievements and extensive international influence.

Hangzhou, Zhejiang, China

Zhejiang University

ZJU 100 Young Professor

Promising young scholars who can independently establish and develop a research direction.

Head of the Thrust of Robotics and Autonomous Systems

Reporting to the Dean of Systems Hub, the Head of ROAS is an executive assuming overall responsibility for the academic, student, human resources...

Guangzhou, Guangdong, China

The Hong Kong University of Science and Technology (Guangzhou)

Head of Biology, Bio-island

Head of Biology to lead the discovery biology group.

BeiGene Ltd.

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Resources Home 🏠
• Try SciSpace Copilot
• Search research papers
• Add Copilot Extension
• Try AI Detector
• Try Paraphraser
• Try Citation Generator
• April Papers
• June Papers
• July Papers

Importance of Peer Review: Every researcher should know

Table of Contents

Peer review is a cornerstone of the scientific world, serving as a rigorous and essential process to ensure the quality and credibility of scholarly work. This systematic evaluation by experts in the field plays a pivotal role in maintaining the standards of academic excellence.

In this article, we will delve into the significance of peer review and explore how it contributes to the advancement of knowledge, the validation of research, and the overall integrity of scholarly communication.

What is Peer Review?

Before understanding why peer review is important, it's crucial to grasp what it is. Peer review is a process where experts in a specific field evaluate and critique a researcher's work before it is published. This process is designed to ensure the work is rigorous and coherent, uses previous research appropriately, and adds to the knowledge in the field.

The peer review process varies across different fields and journals, but the core principle remains the same — to ensure the quality of published research. It is a form of self-regulation by qualified members of the profession within the relevant field.

Why is Peer Review Important?

Let’s understand the importance of peer review in a detailed manner!

Ensures Quality Control

The primary reason why peer review is important is that it acts as a form of quality control for scientific research. It ensures that the research conducted is of high quality, is methodologically sound, and is relevant to the field. This is crucial in maintaining the integrity of the academic record.

Without peer review, there would be no system in place to check the validity of the research or to prevent the publication of substandard or flawed studies. This could lead to a flood of inaccurate or misleading information, which could have serious implications for future research and even public policy.

Promotes Rigorous Research

Peer review encourages researchers to conduct their studies rigorously and ethically. Knowing that their work will be scrutinized by their peers pushes researchers to be meticulous in their methodology, thorough in their literature review, and honest in their conclusions.

It also encourages researchers to continually improve their work. Feedback from peer reviewers can help identify gaps or weaknesses in the research that the author may have overlooked. This can lead to the improvement of the study and contribute to the advancement of knowledge in the field.

Facilitates Scientific Communication

Peer review plays a crucial role in facilitating communication within the scientific community. It ensures that research is communicated in a standard, understandable format and that it meets the norms and standards of the field.

Moreover, the peer review process often involves a dialogue between the author and the reviewers. This exchange of ideas can lead to new insights, spark new research questions, and contribute to the evolution of scientific knowledge.

Contributes to Career Advancement

Peer-reviewed publications are often seen as a mark of quality in academia. They are a critical factor in career advancement, grant applications, and institutional reputation. Therefore, the peer review process plays a significant role in shaping the careers of academics and researchers.

Furthermore, serving as a peer reviewer can also contribute to career development. It allows researchers to stay updated with the latest research in their field, improve their critical thinking skills, and gain recognition from their peers.

Cross-Pollination of Ideas

Through peer review, scholars have the opportunity to share their work with the wider academic community. This fosters a culture of collaboration and allows for the cross-pollination of ideas. Peers provide constructive feedback, suggesting alternative approaches or perspectives that can enrich the research and broaden its impact. This collaborative aspect of peer review enhances the overall quality and depth of scholarly work.

Identification of Bias and Objectivity

Peer review plays a crucial role in identifying and addressing biases in research. Reviewers are tasked with evaluating the objectivity of the study, ensuring that the research is conducted and presented without undue influence or prejudice. This process contributes to the integrity of academic publications, as it helps prevent the dissemination of research that may be skewed or incomplete.

Challenges and Criticisms of Peer Review

Despite its importance, the peer review process is not without its challenges and criticisms. Some argue that it can be slow, prone to bias, and lack transparency. Others point out that it can be a burden on the academic community as it is often unpaid work.

However, despite these challenges, peer review remains the gold standard in academic publishing. It continues to evolve and adapt, with many journals now adopting open peer review or post-publication peer review to address some of these criticisms.

In conclusion, the importance of peer review in maintaining the quality and integrity of academic research cannot be overstated. It serves as a rigorous evaluation process, ensuring that scholarly work meets the highest standards of excellence.

Through the collaborative efforts of peers, the academic community can rely on a system that not only validates research but also contributes to the continual improvement and advancement of knowledge. Peer review is, therefore, an indispensable component of the scholarly publishing landscape, playing a key role in shaping the future of academic excellence.

You might also like

Cybersecurity in Higher Education: Safeguarding Students and Faculty Data

How To Write An Argumentative Essay

Beyond Google Scholar: Why SciSpace is the best alternative

You are using an outdated browser . Please upgrade your browser today !

What Is Peer Review and Why Is It Important?

It’s one of the major cornerstones of the academic process and critical to maintaining rigorous quality standards for research papers. Whichever side of the peer review process you’re on, we want to help you understand the steps involved.

This post is part of a series that provides practical information and resources for authors and editors.

Peer review – the evaluation of academic research by other experts in the same field – has been used by the scientific community as a method of ensuring novelty and quality of research for more than 300 years. It is a testament to the power of peer review that a scientific hypothesis or statement presented to the world is largely ignored by the scholarly community unless it is first published in a peer-reviewed journal.

It is also safe to say that peer review is a critical element of the scholarly publication process and one of the major cornerstones of the academic process. It acts as a filter, ensuring that research is properly verified before being published. And it arguably improves the quality of the research, as the rigorous review by like-minded experts helps to refine or emphasise key points and correct inadvertent errors.

Ideally, this process encourages authors to meet the accepted standards of their discipline and in turn reduces the dissemination of irrelevant findings, unwarranted claims, unacceptable interpretations, and personal views.

If you are a researcher, you will come across peer review many times in your career. But not every part of the process might be clear to you yet. So, let’s have a look together!

Types of Peer Review

Peer review comes in many different forms. With single-blind peer review , the names of the reviewers are hidden from the authors, while double-blind peer review , both reviewers and authors remain anonymous. Then, there is open peer review , a term which offers more than one interpretation nowadays.

Open peer review can simply mean that reviewer and author identities are revealed to each other. It can also mean that a journal makes the reviewers’ reports and author replies of published papers publicly available (anonymized or not). The “open” in open peer review can even be a call for participation, where fellow researchers are invited to proactively comment on a freely accessible pre-print article. The latter two options are not yet widely used, but the Open Science movement, which strives for more transparency in scientific publishing, has been giving them a strong push over the last years.

If you are unsure about what kind of peer review a specific journal conducts, check out its instructions for authors and/or their editorial policy on the journal’s home page.

Why Should I Even Review?

To answer that question, many reviewers would probably reply that it simply is their “academic duty” – a natural part of academia, an important mechanism to monitor the quality of published research in their field. This is of course why the peer-review system was developed in the first place – by academia rather than the publishers – but there are also benefits.

Are you looking for the right place to publish your paper? Find out here whether a De Gruyter journal might be the right fit.

Besides a general interest in the field, reviewing also helps researchers keep up-to-date with the latest developments. They get to know about new research before everyone else does. It might help with their own research and/or stimulate new ideas. On top of that, reviewing builds relationships with prestigious journals and journal editors.

Clearly, reviewing is also crucial for the development of a scientific career, especially in the early stages. Relatively new services like Publons and ORCID Reviewer Recognition can support reviewers in getting credit for their efforts and making their contributions more visible to the wider community.

The Fundamentals of Reviewing

You have received an invitation to review? Before agreeing to do so, there are three pertinent questions you should ask yourself:

• Does the article you are being asked to review match your expertise?
• Do you have time to review the paper?
• Are there any potential conflicts of interest (e.g. of financial or personal nature)?

If you feel like you cannot handle the review for whatever reason, it is okay to decline. If you can think of a colleague who would be well suited for the topic, even better – suggest them to the journal’s editorial office.

But let’s assume that you have accepted the request. Here are some general things to keep in mind:

Please be aware that reviewer reports provide advice for editors to assist them in reaching a decision on a submitted paper. The final decision concerning a manuscript does not lie with you, but ultimately with the editor. It’s your expert guidance that is being sought.

Reviewing also needs to be conducted confidentially . The article you have been asked to review, including supplementary material, must never be disclosed to a third party. In the traditional single- or double-blind peer review process, your own anonymity will also be strictly preserved. Therefore, you should not communicate directly with the authors.

When writing a review, it is important to keep the journal’s guidelines in mind and to work along the building blocks of a manuscript (typically: abstract, introduction, methods, results, discussion, conclusion, references, tables, figures).

After initial receipt of the manuscript, you will be asked to supply your feedback within a specified period (usually 2-4 weeks). If at some point you notice that you are running out of time, get in touch with the editorial office as soon as you can and ask whether an extension is possible.

Some More Advice from a Journal Editor

• Be critical and constructive. An editor will find it easier to overturn very critical, unconstructive comments than to overturn favourable comments.
• Justify and specify all criticisms. Make specific references to the text of the paper (use line numbers!) or to published literature. Vague criticisms are unhelpful.
• Don’t repeat information from the paper , for example, the title and authors names, as this information already appears elsewhere in the review form.
• Check the aims and scope. This will help ensure that your comments are in accordance with journal policy and can be found on its home page.
• Give a clear recommendation . Do not put “I will leave the decision to the editor” in your reply, unless you are genuinely unsure of your recommendation.
• Number your comments. This makes it easy for authors to easily refer to them.
• Be careful not to identify yourself. Check, for example, the file name of your report if you submit it as a Word file.

Sticking to these rules will make the author’s life and that of the editors much easier!

Explore new perspectives on peer review in this collection of blog posts published during Peer Review Week 2021

[Title image by AndreyPopov/iStock/Getty Images Plus

David Sleeman

David Sleeman worked as a Senior Journals Manager in the field of Physical Sciences at De Gruyter.

You might also be interested in

Academia & Publishing

The Impact of Transformative Agreements on Scholarly Publishing

Our website is currently unavailable: cyberattacks on cultural heritage institutions, wie steht es um das wissenschaftliche erbe der ddr eine podiumsdiskussion, visit our shop.

De Gruyter publishes over 1,300 new book titles each year and more than 750 journals in the humanities, social sciences, medicine, mathematics, engineering, computer sciences, natural sciences, and law.

Pin It on Pinterest

Writing a review—the do’s and don’ts

Proceedings b senior editor dr maurine neiman discusses the value of peer review and highlights the different elements to consider when writing a review report..

Preprint and Senior Editor of Proceedings of the Royal Society B , Dr Maurine Neiman , from the University of Iowa, shared her thoughts on peer review and what makes a good review in scientific scholarly publishing.

Why is peer review important for scientists and scientific research?

Peer review is important because scientific peers have the expertise and contextual knowledge that is needed to rigorously and fairly evaluate the extent to which a new scientific paper makes a valid and useful (i.e., publishable) scientific contribution. The process isn’t perfect: humans can never be truly objective, for example, and not all reviewers provide useful or fair reviews. This is where my role as an editor can be especially important: it’s my responsibility to select high-quality reviewers, decide when particular reviews are especially useful or should be taken with a grain of salt, and provide my own independent assessment of the utility and quality of the paper.

As an Editor, what do you think makes a good review of a scientific paper?

There are multiple elements of a high-quality peer review. First , the review should be reasonably objective. Second , the review should be thorough, with the caveat that it can be difficult to find individual reviewers that can rigorously evaluate interdisciplinary studies and papers that feature multiple methodology types (e.g., theory and empirical data). Third , the review should provide a combination of appropriately positive and critical components, and constructive suggestions accompanying the critiques. With respect to the importance of positivity, the editor needs the expertise of the peer reviewers to evaluate the extent to which the paper has strengths as well as weaknesses. Fourth , the review should acknowledge components of the manuscript that the reviewer might not be able to objectively or rigorously evaluate. Finally , the review should provide a realistic perspective on what constitutes a useful contribution to scientific literature, in general, and with respect to the focal journal.

What should reviewers avoid when writing their reviews?

As I mentioned above, reviewers should avoid presenting criticism in the absence of constructive suggestions. As an author and as an editor, it can be very difficult to know how to respond to criticism in the absence of specific suggestions for improvement. As a related point, I think that reviewers should take some care to ensure that their review will come across to the authors as respectful and not overly harsh. No one benefits from an antagonistic review process.

What advice do you have for early career researchers starting out in peer review?

I think that early-career researchers should make an effort to engage in peer review early and often because reviewing is the best way to learn how the peer-review process works and provides valuable insight in how best to craft a scientific manuscript. The best way for junior researchers to start reviewing is often to team up with a graduate or postdoc advisor or other senior mentors. This type of co-review provides a direct but supervised means of gaining experience with review and will help make the junior researcher more visible to their peers, leading to more review invitations in the future. More and more journals are explicitly encouraging peer reviewers to involve students in this way, which I think is a really positive development. I do think that scientists who are new to the peer review process, perhaps as a function of graduate seminar courses focused on ‘paper bashing’, often believe that their main role as a reviewer is to be extremely critical. While this type of review can be useful, new reviewers should remember the importance of speaking to the positive elements of a paper and providing constructive suggestions alongside their critiques.

What are your thoughts on transparency in peer review?

Transparency in peer review is such a complicated issue. While I believe that transparency in science is generally a positive (and important) step forward, I think that our human challenges with objectivity mean that it is important to maintain opacity with respect to the identities of the people reviewing the papers. In other words, anonymous peer review (and, perhaps, in many/most cases, a double-blind peer review process) seems important to implement as a mechanism to maximise objectivity and minimise the negative consequences of implicit and explicit bias. This perspective doesn’t exclude the possibility that the anonymised peer reviews are published alongside the papers (an idea that I like, at least in principle), or that peer reviewers can choose to make their identities public.

If you are interested in reviewing for Proceedings B or any other Royal Society journal, find out about the benefits of reviewing for our journals on our website .

Image credit: Dr Maurine Neiman, University of Iowa

Shalene Singh-Shepherd

Senior Publishing Editor, Proceedings B

Related blogs

Callum Shoosmith

In publishing

The future of peer review in the 17th Century

For Peer Review Week 2023 we explore the Royal Society’s collection of centuries-old referee…

Buchi Okereafor

AI and the future of scholarly publishing (part 1)

The focus for this year's Peer Review Week is 'Peer Review and the Future of Publishing'. We speak…

AI and the future of scholarly publishing (part 2)

Matt Hodgkinson, Research Integrity Manager at the UK Research Integrity Office, gives us an…

Email updates

We promote excellence in science so that, together, we can benefit humanity and tackle the biggest challenges of our time.

Subscribe to our newsletters to be updated with the latest news on innovation, events, articles and reports.

What subscription are you interested in receiving? (Choose at least one subject)

What is peer review?

From a publisher’s perspective, peer review functions as a filter for content, directing better quality articles to better quality journals and so creating journal brands.

Running articles through the process of peer review adds value to them. For this reason publishers need to make sure that peer review is robust.

Editor Feedback

"Pointing out the specifics about flaws in the paper’s structure is paramount. Are methods valid, is data clearly presented, and are conclusions supported by data?” (Editor feedback)

“If an editor can read your comments and understand clearly the basis for your recommendation, then you have written a helpful review.” (Editor feedback)

Peer Review at Its Best

What peer review does best is improve the quality of published papers by motivating authors to submit good quality work – and helping to improve that work through the peer review process. 

In fact, 90% of researchers feel that peer review improves the quality of their published paper (University of Tennessee and CIBER Research Ltd, 2013).

What the Critics Say

The peer review system is not without criticism. Studies show that even after peer review, some articles still contain inaccuracies and demonstrate that most rejected papers will go on to be published somewhere else.

However, these criticisms should be understood within the context of peer review as a human activity. The occasional errors of peer review are not reasons for abandoning the process altogether – the mistakes would be worse without it.

Improving Effectiveness

Some of the ways in which Wiley is seeking to improve the efficiency of the process, include:

• Reducing the amount of repeat reviewing by innovating around transferable peer review
• Providing training and best practice guidance to peer reviewers
• Improving recognition of the contribution made by reviewers

Visit our Peer Review Process and Types of Peer Review pages for additional detailed information on peer review.

Transparency in Peer Review

Wiley is committed to increasing transparency in peer review, increasing accountability for the peer review process and giving recognition to the work of peer reviewers and editors. We are also actively exploring other peer review models to give researchers the options that suit them and their communities.

• Find out more
• Members area
• Members registration

What is Peer Review and why is it important?

Posted on 22 September 2022

We asked Emma Soneson , postdoctoral researcher at the University of Oxford, about Peer Reviewing.

What is Peer Review, and why is it important?

Peer review is a fundamental part of scientific publishing whereby individuals with relevant expertise who were not involved in a particular piece of research critically appraise others’ work to ensure that it meets standards for ethical conduct, quality, and rigour. In fact, peer review is such a fundamental part of the research process that it’s hard to imagine a world without it!

In the field of child and adolescent mental health, decisions about policy and practice rely on having high-quality evidence readily available. As peer reviewers, we help in this decision-making process by assessing the quality and implications of research. Without peer review, it would be far more difficult (and time-consuming) for clinicians, policymakers, educators, researchers, and members of the public to evaluate research in our field. Peer reviewers therefore play a critical role in helping appraise the evidence and ensure that it’s acceptable for publication and dissemination.

The theme of this year’s Peer Review Week is ‘Research Integrity: Creating and Supporting Trust in Research’. How do you think we can create and support trust in research through the peer review process?

If we’ve learned anything over the past few years, it’s that rigorously conducted and thoroughly peer-reviewed research is essential in promoting and protecting people’s health. Unfortunately, we’ve also seen how poor-quality research that has ‘slipped through the cracks’ in terms of peer review can have serious and wide-reaching consequences. When done properly, peer review can help build trust in science by serving as a ‘quality control’ measure that ensures that research – and especially the research that hits the front page of the newspapers and/or forms the basis of health policy and practice – has been conducted to a high standard. Rigorous peer review can lend credibility to research and ensure that decision-making is based on a strong foundation of high-quality evidence.

As an early career researcher, what influenced you to be a peer reviewer and what impact has it had on your own work?

As is likely the case with many early career researchers, I was initially nervous that I wouldn’t be a ‘good enough’ peer reviewer. After all, how could I, as a new researcher, judge the work of others who have been in this field for decades? However, I knew that participating in the peer review process was an important part of being a researcher and in ‘giving back’ to the scientific community, and so was eager to contribute.

Luckily, the first review I ever wrote was in partnership with my supervisor, who helped guide me through the process. When I sent back pages of line-by-line comments, he clarified that it’s more useful to focus on a few overarching issues rather than to worry about things like grammar and formatting. Having his guidance was immensely helpful for this first review (and I really hate to think how the authors would have reacted after receiving five pages of comments!) Since that first review, I’ve reviewed for nine journals and have become a lot more confident – and hopefully more useful! – as a reviewer.

While I’m glad to be contributing to improving the science behind child and adolescent mental health, I also gain a lot from peer review myself. Through critically assessing others’ work, I’ve learned what reviewers are looking for and can therefore make sure that my own papers address those points. I’ve also learned a lot about our field this way! Doing peer reviews helps make sure that I stay on top of the most recent research in my area, which helps me to develop my own research ideas and make sure that my work is impactful. Finally, peer review helps expose me to new methods that I can use in my own research. Just recently, I reviewed a systematic review that used a style of quantitative synthesis that I’d never seen before but that I will definitely consider using moving forward!

What is your approach when it comes to peer reviewing?

The first step for me is always determining whether I’m a suitable reviewer for the paper. Any time I get an invitation, I always read the abstract to make sure that I am confident enough in the content area and methods used to be able to write a useful, well-informed review.

If I feel it’s a good fit, I start by reading the entire paper through once without writing a single comment. This helps me get a sense of the bigger picture of the article and its narrative before focusing in on the detail (this is something that I’ve changed since I first started reviewing, and I highly recommend this strategy!) On the second readthrough, I start making notes to myself about potential issues and check the methods against the authors’ pre-registration, if available. In this iteration, I always try to comment on objective issues, rather than personal preferences (e.g. ‘This paragraph is technically correct, but I would have worded it differently’). It’s important to remember that different authors have different styles, and your job as a reviewer is to assess the merit of their work, not to say how you would have written the paper!

When I finally write up the review for the authors and editors, I always structure it in a way that assesses both the strengths and weaknesses of the paper, as from an author perspective it’s never nice to receive a page full of solely-critical comments! I put a lot of care into how I phrase all of my points and make sure to offer potential suggestions for improvements and solutions rather than just raising issues. My general motto is to never write a review that I wouldn’t be happy to read aloud to the authors!

Can you give us a few examples of what you think makes a good peer reviewer?

Having been on the receiving end of many peer reviews, I’ve learned a lot about what makes a good review! Regardless of the final recommendation (i.e. to accept/revise/reject), it’s always useful when a reviewer provides constructive criticism on how the paper can be improved. As I mentioned above, nothing is more disheartening than receiving a laundry list of negative comments. It’s so much more useful to receive suggestions on how to improve! For example, can the methods be clarified? Can the recommendations for practitioners be made more impactful? These are the types of suggestions that can help make the paper – and science as a whole – better. Relatedly, I find it useful to hear what was good about a paper. This helps me understand how to progress my research moving forward and to ensure that it’s useful and relevant to the field.

I also appreciate reviews that are specific. Again, regardless of the final recommendation you make, it’s always helpful to give the precise details and reasoning behind your decision. As an author, I’ve been on the receiving end of the ‘single sentence review’ (in my case, an unfavourable assessment of our paper), and I can say with complete certainty that this is not helpful to anyone! It doesn’t help the editors to make an informed decision about publication, it doesn’t help the authors to improve their paper, and it doesn’t help to move research forward. Authors spend a lot of time conducting research and writing it up for submission, and they deserve more than a single sentence in return!

Last but not least, a good review is one where the reviewer has acknowledged any limitations they may have. Are you an expert in a topic area, but not 100% sure about whether the statistical methods used were appropriate? Your review can still be incredibly valuable, but do make sure to acknowledge your uncertainties about the methods! This can help the editor to search for a reviewer with the specific expertise you’re lacking. None of us are experts in everything, and acknowledging our limitations is an important part of enabling a rigorous review process!

Why do you think early career researchers should get involved in peer reviewing, and how can they go about this?

There are so many benefits of peer reviewing, especially for early career researchers. Peer review is valuable for improving your own research through learning about how papers are assessed as well as keeping on top of your topic area and innovative methodology!

Given my own experiences, I’d always recommend doing your first peer review with supervision from someone who’s done them before. This can help make sure you’re on the right track with your comments and can boost your confidence if you’re feeling nervous about completing your first review. After this, you can sign yourself up as a reviewer for journals you like, so that they know they can contact you as a reviewer. By the time you’ve written a few papers as corresponding author, don’t be surprised if the journals contact you directly!

But, as with all things, it’s important to remember that it’s a balance. At this point, I receive an average of two peer review invitations a week. It’s important not to take on too many, as doing a good peer review takes time. Instead, I recommend focusing on one review at a time, to ensure that you have the time and energy to do a thorough job.

This is helpful. Thank you.

Your email address will not be published. Required fields are marked *

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Why Use Peer Reviewed Articles in your Research?

What are peer reviewed articles.

Peer reviewed, or scholarly, articles are written by experts and are read and reviewed by other scholars in the field.  Their comments and suggestions are incorporated into the article before it is published in a scholarly journal. This process is known as peer review and helps check the accuracy and validity of the research as well as the article’s overall quality.

These articles are the main way scholars communicate with each other about their research findings.  They are like a conversation between scholars on a particular topic.  Sometimes scholars agree, and sometimes they don’t!  And just like a conversation, it is all about finding the pertinent information and moving the conversation forward.

How do articles get peer reviewed? What role does peer review play in scholarly research and publication?

Why use Peer Reviewed Articles?

Peer reviewed articles are considered the gold standard source type for academic research.  They are written by researchers, or experts, on the topic, so it takes some of the guess work out of wondering if you should use them or not.

In peer reviewed articles, you will find:

• findings from original research
• detailed information about a narrow aspect of your topic
• expert language
• charts or graphs
• a list of references, or sources, the author used

Your Research Journey at Portland State University Library Copyright © 2020 by Amy Stanforth is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Science-Based Medicine

Exploring issues and controversies in the relationship between science and medicine

The Importance and Limitations of Peer-Review

Peer-review is a critical part of the functioning of the scientific community, of quality control, and the self corrective nature of science. But it is no panacea. It is helpful to understand what it is, and what it isn’t, its uses and abuses.

When the statement is made that research is “peer-reviewed” this is usually meant to refer to the fact that it has been published in a peer-reviewed journal. Different scientific disciplines have different mechanisms for determining which journals are legitimately peer-reviewed. In medicine the National Library of Medicine (NLM) has rules for peer-review and they decide on a case by case basis which journals get their stamp of approval. Such journals are then listed as peer-reviewed.

The basic criterion is that there is a formalized process of peer-review prior to publication – so this presents a barrier to publication that acts as a quality control filter. Typically, the journal editor will give a submitted paper to a small number of qualified peers – recognized experts in the relevant field. The reviewers will then submit detailed criticism of the paper along with a recommendation to reject, accept with major revisions, accept with minor revisions, or accept as is. It is rare to get an acceptance as is on the first round.

The editor also reviews the paper, and may break a tie among the reviewers or add their own comments. The process, although at times painful, is quite useful in not only checking the quality of submitted work, but improving the quality. A reviewer, for example, may point out prior research the authors did not comment on, or may point our errors in the paper which can be fixed.

It is typical for authors to submit a paper to a prestigious journal first, and then if they get rejected to work their way down the food chain until they find a journal that will accept it. This does not always mean that the paper was of poorer quality – the most prestigious journals have tons of submissions and can pick and choose the most relevant or important studies. But sometimes it does mean the paper is mediocre or even poor.

The limitations of Peer-Review

It is important to realize that not all peer-reviewed journals are created equal. Small or obscure journals may follow the rules and gain recognized peer-reviewed status, but be desperate for submissions and have a low bar for acceptance. They also have a harder time getting world-class experts to review their submissions, and have to find reviewers that are also farther down the food chain. The bottom line is that when a study is touted as “peer-reviewed” you have to consider where it was reviewed and published.

Even at the best journals, the process is only as good as the editors and reviewers, who are people who make mistakes. A busy reviewer may give a cursory read through a paper that superficially looks good, but miss subtle mistakes. Or they may not take the time to chase down every reference, or check all the statistics. The process generally works, and is certainly better than having no quality control filter, but it is also no guarantee of correctness, or even the avoidance of mistakes.

Peer-reviewers also have biases. They may be prejudiced against studies that contradict their own research or their preferred beliefs. They may therefore bias the published studies in their favored direction, and may be loath to give a pass to a submission that would directly contradict something they have published. For this reasons editors often allow authors to request or recommend reviewers, or to request that certain people not be asked to be reviewers. Each journal has their own policy. Sometimes an editor will specifically use a reviewer that the authors request not be used, thinking they may be trying to avoid legitimate criticism.

The process can be quite messy, and full of politics. But in the end it more or less works. If an author thinks they were treated unfairly by one journal, they can always go to another or they can talk directly to the editor to appeal a decision and try to make their case.

Perhaps the biggest weakness of peer-review, however, is when an entire discipline of peers is lacking in some fundamental way. For example, there are now many journals dedicated to so-called “alternative medicine” ( CAM ).  The editors of such journals tend to have a pro- CAM bias, and they find reviewers with a pro- CAM bias. So pretty much any pro- CAM article can get published. Some have enough ideological friends at the NLM that they can get approved as peer-reviewed, despite glaring biases in their editorial policy.

Post Publication Peer Review

The term peer-review is sometimes used to refer to the fact that papers are read and reviewed by the broader scientific community once they are published. However, this post-production review should not be confused with “peer-reviewed” and that term should not be used to refer to post-publication review, to avoid confusion.

The process, however, is even more critical to quality control in science. Now, instead of one editor and 2-3 reviewers looking at a study, dozens or hundreds (maybe even thousands) of scientists can pick over a study, dissect the statistics and the claims, bring to bear knowledge from related areas or other research, and provide detailed criticism. This is the real “meat grinder” of science. Hundreds of reviewers are more likely to find problems than the few pre-publication reviewers. Arguments can be tested in the unforgiving arena of the scientific community, weeding out bad arguments, honing others, so that only the best survive.

Here is the bottom line – peer-review is a necessary component of quality control in science, but is no guarantee of quality, and you have to know the details of the journal that is providing the peer-review.

Founder and currently Executive Editor of Science-Based Medicine Steven Novella, MD is an academic clinical neurologist at the Yale University School of Medicine. He is also the host and producer of the popular weekly science podcast, The Skeptics’ Guide to the Universe , and the author of the NeuroLogicaBlog , a daily blog that covers news and issues in neuroscience, but also general science, scientific skepticism, philosophy of science, critical thinking, and the intersection of science with the media and society. Dr. Novella also has produced two courses with The Great Courses , and published a book on critical thinking - also called The Skeptics Guide to the Universe .

View all posts

• Posted in: Clinical Trials , Science and Medicine

Posted by Steven Novella