Case Study: Should you listen to a peer reviewer?

ORI will soon release a series of RCR case studies edited by Dr. James Dubois of St. Louis Univerity.  The creation of the case studies was funded through ORI's RCR Resource Development program and involved a team of nearly 20 writers, contributors, and reviewers.  These well-crafted case studies, along with role playing scenarios, will be available for instructors to incorporate into their institutions' RCR training programs. Below is one of many case studies that will be available.
Getting a Fair Shake?
Gail is an Associate Professor who has published extensively and rightly considers herself one of the leading experts in her field. She knows the drill for developing and submitting a proposal for funding. Her most recent proposal is based on considerable prior research that she and colleagues have conducted, and it employs methodology that she knows will produce valid findings.
Gail is anxious about her proposal being reviewed and funded. Prior to submitting, she ran it by a network of capable colleagues with whom she works collaboratively to develop and critique ideas. They all share the same dedication to medical research with important long-term implications for public health, as well as short-term implications for federal policies and funding.
When Gail finally receives the funder’s evaluation, she is happy to see that she received a high score on her proposal. However, the score is below the cut-off point at which it can be funded. Gail turns to see what objections and concerns the reviewers had with her proposal. She plans to revise and resubmit for the next funding cycle.
Gail is flabbergasted to read what the reviewers said. In their feedback, they find that her methods are “flawed”. And they suggest changes to the study design that Gail is certain would significantly compromise the quality of the science as well as its impact on the development of her field. They clearly did not understand her approach, which is highly novel.
Gail is considering re-submitting the proposal employing the reviewers’ suggested methodology, then actually doing the study as she proposed it in the first place, explaining that “pilot testing” indicated that her methodology would work best.
However, Gail worries that this is deceptive. And who knows if the same reviewers would be evaluating her resubmission? Perhaps she is better off proposing her original “flawed” methodology—with a direct response to reviewers—in hopes that she will receive new reviewers or that her reviewers will be unusually open minded. Gail has two months before the revised proposal is due. She feels caught between a rock and a hard place.
What should Gail do?



I agree with the previous post. If reviewers misunderstand me, it is almost always because I failed to describe my plans adequately. This case study seems to fall within this category because apparently all the reviewers misunderstood. Alternately, "Gail" could incorporate the reviewers' suggestions as an alternative if her approach fails. Who knows, they could be correct? In any event, the difference of opinion does need to be acknowledged, but to simply change her approach to appease the reviewers is likely to backfire if for no reason other than the same reviewers may not review the revision.

The situation is completely different if the reviewer or a reviewer understands but disagrees with the applicant. This situation can occur when two or more schools of thought about mechanism may exist, and the proposal has been reviewed by someone from the other camp. I believe the best way to handle this situation is to devise a head-to-head test of both competing theories, or at the least acknowledge the difference of opinion and state that although one is aware of different theories, the one being proposed is the favored, and if it is disproved, then at least it will be taken off the table.


Should You Always Be Responsive to Peer Review? ABSOLUTELY!

The ugly fact is that Reviewers often have an unsatisfactory base of knowledge to review groundbreaking applications, and basically lack the tools to understand the more innovative proposals.

However, these same reviewers have huge egos.

There are no benefits to arguing with the review panel, even when Reviewers are patently wrong. Over the years of submitting grants to the NIH, this one truth emerged rather quickly: Reviewers do not appreciate any contradiction. I am yet to hear of a single incident where the applicant successfully argued with the reviewers and was awarded the funds to conduct the disputed studies. It is far easier to acquiesce to the reviewers suggestions, and after receiving funding, conduct the suggested experiments, and armed with the empirical knowledge that the reviewer panel got the wrong end of the stick, go back to the correct originally proposed research design. The cost of this approach is some wasted time and money, however, on the plus side, you have funds to conduct important studies.


Absolutely! She must appease the reviewer. This is the only way to get funded. Reviewers are vengeful and any contradiction is automatically viewed as a good reason to further reduce the score.

I agree with the first comment.  Even if you think that you have been crystal clear, you need to take responsibility for a miscommunication.  If the proposed methodology is the best approach, you should stick to your guns, but re-write until your reasoning is apparent to even a novice to your field.  The original reviewers must have liked the proposal a great deal, or the score wouldn't have been as high as it was.  


The above response is spot-on.   As grant writers, we are teachers, self-advocates, and our own and only sales person.    It is our task to convince the reader, period.  If the reader doesn't "get it," we failed to convey our message.   The reader can be less versed in the field, even less intelligent.  Doesn't matter.  Complaining about such is like saying that none of the students in our classes "get it" even though we are the leader....    My mentor taught me the magic phrase for response/resubmission, "We were unfortunately unclear in our [presentation/explanation/prior description]..."  and from there to go on to add a figure, and make a table or bullet points or whatever it takes to make it clear to an eight grader, but never bark.  The reader essentially holds the funding in his or her hands, so treat it like a job interview.


The critical phrase here is "They clearly did not understand her approach, which is highly novel".To put it mildly, she needs to show some humility. She needs to seriously consider that the reason the reviewers did not understand her approach, is she did a bad job of explaining it. Instead of running the proposal past her colleagues who are already familiar with the work, she should try rewriting it for an audience of novices. To respond with anything other than an apology for not being clear, will almost guarantee the resubmission gets triaged.Yeah yeah bad response for the original topic, but these case studies need to be about the real world, and in the real world, grant writers who complain that reviewers just don't "get it", are the ones who get fried.

I agree with the previous comment.  If the reviewers don't "get it," it is always the applicant's fault...she should explain it better instead of whining.  Reviewers are almost always a bit outside of the specific field of the applicant's expertise and need more explanation than one might give a collaborator.  

And, they are doing a lot of work for virtually no pay, so the applicant should make it easy for them, even give them some enjoyment in reading the application.  I reviewed a grant once that said in the usual tedious section about resources that his office of so many square feet had a great view of a  parking lot.  Humor helps.


The worst approach would be the bait-and-switch: submit a proposal for method B and then actually use method A. 

The reviewers "suggest changes to the study design that Gail is certain would significantly compromise the quality of the science as well as its impact on the development of her field." 

I'm pleased Gail is proposing a novel approach. Still, I'd like her to be more open-minded to alternate points of view. Why can't she objectively address a professional critique from peers? Possible answers: she's under pressure to raise grant money, she has a healthy ego, and we live in a winner-take-all society. 

If the root problem isn't her poor communication (see comment above) and instead rests on true scientific differences, perhaps there's value in trying both methods A and B and comparing the results. Based on the facts though, I wouldn't trust Gail to make the comparison since she seems too wedded to method A.  

She may be barking up the wrong tree with her novel approach. Gail alone does not define quality science. 

Even if "her" field (as if she owns it) eventually suffers negative impact, science overall may benefit. 


You do have choices in numbers. Peer review is a good process in the developement in research. However, you are not confind to just one reviewer. Some studies may require more than one reviewer to truely help the researcher best with ethical or moral ideations.


The commentator above says "........she needs to show some humility". Why should the scientist be considered "proud" or "not humble" just because she is presenting her INSPIRATION that may lead to some "new" scientific finding? This attitude of projecting one's personal inadequacies as being the fault of of another is a dangerous practice in the scientific community. Essentially, inorder to arrive at a certain finding one must apply a plausible, clear, and reprodicible method that is in harmony with the age-old standard, the socalled "scientific method". Once this criterion is objectively satisfied, there should have been no real problem in funding the work of [a ] scientist--especially someone of well established institutional experience and publication history. Why "block"another's progress because some other researcher blames the proposal writer forhis personal lack of understanding as being the lack of "humility"?

Many doctoral programs use preparation of a grant proposal and response to critique as a qualifying examination.  Students could encounter the same situation.  In using this example, the relevance to the student in the near term and not just later in an academic career (and most students do no end up in academics) should be drawn.

Add new comment

Source URL: https://ori.hhs.gov/blog/case-study-should-you-listen-peer-reviewer