Assessing quality
ORI Introduction to RCR: Chapter 10. Peer Review
Journal editors, grant administrators, and others rely on peers to assess the quality of proposed and published research. Some parts of an application or manuscript can be checked fairly easily. Are the calculations correct? Is the method that has been used or proposed appropriate? Do the reported results support the conclusions? Other parts are more difficult to confirm. Have the data been accurately recorded and reported? Were the experiments run? Did the subjects give consent? Do the articles cited in the references and bibliography contain the information they are said to contain?
Peers who are asked to make judgments about the quality of a proposed or completed project must do their best to determine whether the work they have been asked to review is internally consistent and conforms to the practices of their field of research. This certainly includes:
- assessing whether the research methods are appropriate;
- checking calculations and/or confirming the logic of important arguments;
- making sure the conclusions are supported by the evidence presented; and
- confirming that the relevant literature has been consulted and cited.
At the very least, peer reviewers should be expected to assess whether the manuscript or proposal under review makes sense and conforms to accepted practices, based on the information presented.
Research that conforms to accepted practices can still have problems. Research quality can be compromised by:
- careless mistakes made in reporting data and/or listing citations;
- the deliberate fabrication and falsification of data;
- improper use of material by others (plagiarism);
- inaccurate reporting of conflicts of interest, contributors/authors; and
- the failure to mention important prior work, either by others or by the researcher submitting a paper for publication.
However, how much peer reviewers can or should do to detect these and other deceptive or sloppy practices remains subject to debate.
There are limits to the amount of checking that is both reasonable and practical. Unless given permission to do so, reviewers should not discuss the work they are reviewing with the authors. In many cases, reviews are “blind” (no author identification), so reviewers could not check with authors even if they wanted to. In addition, it is not reasonable to expect reviewers to check every reference and detail. The fact remains, however, that peer reviewers frequently miss problems that might have been detected had the reviewer checked a little more carefully.
If you agree to serve as a peer reviewer, remember that you have essentially been asked to provide your stamp of approval for someone else’s work. In such circumstances, it is wise to do your homework. Do not give your stamp of approval too easily.