Grant proposals and manuscripts submitted for publication are routinely reviewed by peers, that is, by other researchers with expertise on the topic being addressed and the methods used. Peer review plays several roles in science. It may lead to improvements in study designs, data analysis, or the articulation of results; it provides quality assurance by acting as a mechanism for rejecting proposals and articles that do not meet quality standards; and it provides a device for distributing “scarce goods” in research such as grant dollars and space in premium journals.^37,38

As early as 1937, the National Cancer Institute began requiring peer review of research funding proposals. Today all major federal agencies use peer review to assess the quality of proposals and the expertise of the listed personnel.⁹ Peer review also commonly occurs when investigators apply for promotions or when their work is being used to shape policies or inform the outcomes of legal cases.¹³

In 2010, NIH issued a notice articulating a general expectation of funded researchers to serve as reviewers:

“The NIH peer review system is the foundation of the NIH extramural research enterprise, and its continued excellence depends on our ability to recruit and retain the most accomplished, broad-thinking and creative scientists and experts to serve as peer reviewers. … Therefore, the NIH calls upon investigators who have received research grant funding from the NIH to serve on NIH study sections and advisory groups when invited to do so.”³⁹

As part of the evaluation process, reviewers use a variety of criteria to determine the quality of a grant proposal. These are typically stipulated by the funding agency. For instance, NIH asks reviewers to score proposals according to significance (does the project break important ground?); investigators (are the researchers well suited to the project?); innovation (does the proposal use novel theoretical concepts, approaches, methodologies, etc.?); approach (is the approach to the project sound?); and environment (does where the work is done make a difference?).⁴⁰

Peer reviewers are also asked to evaluate animal/human subject sampling plans, biosafety, human subjects’ protections, and budgets. Other agencies and foundations provide reviewers with guidelines that reflect their priorities for funded projects.

To be effective, peer review must be: timely, thorough, constructive, free from personal bias, and respectful of the need for confidentiality.¹³ Reviewers need to have sufficient expertise in the area being evaluated to serve as an effective judge of its merit, and dedicate adequate time to reading material carefully and providing thoughtful comments in a timely manner. Reviewers are typically asked to identify both strengths and weaknesses of projects. Reviewers are expected to maintain the confidentiality of materials they review; they are prohibited from distributing the materials (at least in the absence of rare, special privileges), and frequently they are asked to delete or destroy materials following review.

The matter of bias is particularly difficult to manage. Studies suggest that some peer reviewers are biased and that there is low agreement of opinions on the same grant proposal or potential publication.^41-43 Biases may arise from competition, institutional membership, personal feuds, and theoretical disagreements.⁴⁴ Blind peer review—that is, masking the identity of reviewers—may facilitate honest and frank evaluation of projects; however, it may also diminish public accountability for reviews. Some journals have moved to open peer reviews. Most funding agencies and journals also expect reviewers to disclose conflicts of interest and to recuse themselves from the review of work by institutional colleagues and research collaborators. Finally, reviewers should recuse themselves if they are unqualified to review a particular project or publication.¹³

In short, peer review in the sciences remains an important but imperfect process that aims to promote objective, reliable, and quality research. NIH has committed to ongoing review and evaluation of its review processes in response to data and feedback from investigators and reviewers.

The Cases and Role Play

The cases and role play in this chapter offer the opportunity to work through some common scenarios that peer reviewers regularly face—as well as challenges that researchers face as they grapple with reviewer feedback.

• Case One: A seasoned scientist and peer reviewer for a notable journal experiences difficulty with a newly appointed journal editor who believes that articles are being published even in the face of reviewer criticism.
• Case Two: A professor submits a grant proposal for peer review and is discouraged when she receives a critique from one of the reviewers, who suggests altering the study design, which the professor believes will compromise the quality of the science.
• Case Three: A researcher in a highly specialized area who received conflicting reviews on a complex grant proposal later finds out that the individual she suspected of ruining her funding chances has published on the exact study design and theory she had submitted.
• Role Play: A peer reviewer receives two papers from different journals to review that are remarkably similar, and wonders if this is duplicative publishing or a dispute over publishing rights?