Judging importance
ORI Introduction to RCR: Chapter 10. Peer Review
In addition to quality, peer reviewers are also asked to make judgments about the importance of proposed or published research. They are asked to answer questions such as:
- Assuming a researcher could carry out a proposed research project, is it important to do so?
- Are these research results important enough to publish?
- Has a researcher made important contributions to a field of study?
- Is this evidence important enough to be used in setting policy?
Along with quality, judgments about importance essentially determine which research is funded or published and which researchers are hired and relied upon for advice.
Peer reviewers do not always make judgments about importance with an open mind. Studies have shown that they can be swayed by:
- the stature of the researcher who conducted the research or the institution at which the research was conducted;
- country of origin;
- a preference for one research method over another, e.g., a clinical versus a laboratory approach; and
- the outcome of the studies under review.
For the most part, these factors should not have a bearing on judgments about importance and yet they do. Each has been shown to influence the judgments peer reviewers make about the publication of research results (see articles by Callaham, Cho, Dickersin, Godlee, Jadad, and Link, Additional Reading).
There is no simple solution to the problem of bias in peer review. Peers frequently are not of one mind about what is or is not important. One reviewer may feel that a field of research should move in one direction, a second in an entirely different direction. Often, it takes time and more research to find out whether a line of investigation or a particular set of findings is important. Nonetheless, researchers can take steps to lessen the impact of bias on their judgments and to help others judge for themselves whether a researcher has biases.
One way to lessen the impact of bias is to write transparent reviews. By “transparent” is meant laying out clearly for anyone reading the review how it was prepared, the literature that was used, and the reviewer’s own possible biases. If reviewers fully and carefully explain how their judgments about importance were made, others can assess whether they want to accept those judgments.
A second way that has been proposed to lessen the impactof bias is to eliminate anonymous reviews. Some argue that this would lessen the candor and rigor of reviews; others that it would make reviewers more accountable. For the present, most reviews are anonymous, which places the burden for fairness on the reviewer. If you have strong feelings about a person or particular line of investigation, tell the person who asked you to do the review and consider whether you can, in fact, provide an impartial assessment.