Study of Inquiry Reports Not Submitted to ORI
July 1998
EXECUTIVE SUMMARY
ORI conducted a content analysis of 21 inquiry reports that were not submitted to the ORI because an investigation was not recommended and ORI had not previously requested the report. The study addressed the following questions:
(1) Were the inquiries being reported by institutions on the Annual Report subject to PHS jurisdiction?
(2) Did the institutions sufficiently document the rationale for deciding that an investigation was unwarranted?
(3) Did the conduct of the inquiries comply with the regulation?
(4) Is more technical assistance needed in the conduct of inquiries?
This study demonstrated that more than half of the institutional inquiry reports that were not submitted to ORI were significantly deficient. Fifty-seven percent of the reports did not contain the information required to establish PHS jurisdiction. Thirty-three percent contained information on no more than four of the nine criteria used to determine whether an investigation was warranted and another 28 percent were marginal, covering only five criteria. Seventy-one percent provided information on only three or fewer criteria for determining compliance with the regulation. And finally, 57 percent of the reports did not contain the detailed information required to justify the decision that an investigation is unwarranted. The analysis was based solely on the content of the submitted reports. Additional information supporting the decision that an investigation was unwarranted may exist in other documents that were not submitted.
These findings suggest that more technical assistance is needed in the following areas:
- Establishing PHS jurisdiction.
- Interpreting the PHS definition of research misconduct, especially plagiarism and "other practices."
- Conducting a thorough, objective, and competent inquiry.
- Preparing inquiry reports in a manner that demonstrates that a decision not to proceed to an investigation is warranted.
- Reporting information that supports compliance with the regulation.
INTRODUCTION
The Federal regulation, 42 C.F.R. Part 50, Subpart A, requires institutions to establish an administrative process for addressing allegations of rresearch misconduct involving research for which Public Health Service (PHS) funds were applied for or awarded. In the conduct of an inquiry, the regulation requires institutions to (1) employ appropriate expertise, (2) protect against conflicts-of-interest, (3) provide the respondent with an opportunity to comment on the allegation and findings, (4) give the respondent a copy of the inquiry report, (5) maintain confidentiality, (6) document reasons for exceeding the 60-day standard for conducting the inquiry, (7) protect the position of the whistleblower, (8) restore the reputation, if appropriate, of the respondent, (9) take interim administrative actions to protect Federal funds, and (10) maintain sufficient documentation for 3 years to permit later assessment of the reasons for determining that an investigation is not warranted.1The regulation also specifies that an institution shall notify the Office of Research Integrity (ORI) of an allegation only when an investigation is warranted. ORI may request information on other inquiries as it deems appropriate.
ORI has oversight responsibility for the implementation of the Federal regulation. Through the Annual Report on Possible Research Misconduct ORI has gathered information regarding inquiries that did not result in an investigation.2This situation raised the following questions: (1) Were the inquiries being reported by institutions on the Annual Report subject to PHS jurisdiction? (2) Did the institutions have sufficient documentation for deciding that an investigation was unwarranted? (3) Did the conduct of the inquiries comply with the regulation? (4) Is more technical assistance needed in the conduct of inquiries? This study was designed to address those questions.
METHODOLOGY
A review of the 1994 and 1995 Annual Reports indicated that 21 institutions had conducted inquiries for which reports were not submitted to ORI because the inquiries did not proceed to an investigation, ORI had not requested the report, or the institution had not voluntarily submitted it. Because there was no finding of misconduct in these cases, protecting the identity of the respondent was paramount. The concern for confidentiality affected the data collection and analysis phases of the study. The institutions were requested to submit the "final report and documentation" for each inquiry. (Attachment 1.) Almost all of the cooperating institutions redacted the reports before submitting them; ORI redacted the others. The material was submitted in a plain envelope with no return address. Transmittal letters were sent under separate cover informing ORI that the report had been submitted. The envelopes and letters were destroyed upon receipt. Following its analysis, ORI destroyed the list of organizations contacted for the study.
Sixteen institutions submitted a total of 22 reports.3Two institutions submitted more than one report. Some participating institutions replied that they could not participate in the study if they had to submit additional documentation because of the redaction burden. Five institutions that declined to participate in the study did so for a variety of reasons. One institution reported that ORI could use the inquiry reports already submitted by the institution for the study. This institution also asserted that the identity of the respondent could not be adequately protected and cited the administrative burden posed by the redaction request. Another institution was concerned about protecting its identity and questioned whether reports submitted by a small number of institutions would provide useful information. The third institution reported that redacting the report would not protect the privacy of the respondent because of the unique research involved in the case. The remaining two institutions described the handling of their allegations and ORI determined that one allegation was dismissed at the preliminary assessment stage as frivolous while the other was dismissed in a lawsuit.
Two independent coders reviewed the reports. Intersubjective reliability was 85 percent. Coding disagreements were discussed by the coders and resolved. The coding form used was a modified version of a compliance review form (
An inquiry is based on an allegation of research misconduct. Research misconduct is defined by the PHS as "fabrication, falsification, plagiarism, or other practices that seriously deviate from those that are commonly accepted within the scientific community for proposing, conducting, or reporting research. It does not include honest error or honest differences in interpretations or judgments of data."4
To establish PHS jurisdiction, two criteria must be met. First, the allegation must fall within the PHS definition of research misconduct. Second, the allegation must involve research for which PHS funds have been requested or awarded.
Information needed to establish PHS jurisdiction was provided in only 43 percent of the reports. (See Table 1.) Forty-eight percent of the reports did not contain allegations that fell within the PHS definition nor provide evidence of PHS funding. The remaining 9 percent only fulfilled one of the criteria.
Table 1: Number of inquiry reports by PHS jurisdiction.
| Jurisdiction | Reports |
|---|---|
| Jurisdiction established | 9 |
| Jurisdiction not established | 12 |
| Allegation/No PHS funding | 1 |
| PHS funding/No allegation | 1 |
| No allegation/No PHS funding | 10 |
|
Total |
21 |
The allegations that came under PHS jurisdiction were falsification, fabrication, fabrication/falsification, and plagiarism. The most frequent type of allegation that did not meet the PHS definition of research misconduct was authorship disputes. These disputes were presented frequently as plagiarism, but are not considered by ORI to fall within the PHS definition of plagiarism. Disagreements over order of authorship or over rights to publish or use ideas among collaborators would fall within this category.5The other allegations - sloppy research practices, protocol violations, patent disputes, sabotage of lab resources - may have been viewed as meeting the definition under the "other practices" clause.
INVESTIGATION WARRANTED
Nine criteria were used to determine whether an inquiry report contained sufficient information to determine whether an investigation was warranted. (See Table 2.) Each criterion impacts on the conduct of a thorough, objective, and competent inquiry. Expertise and conflicts of interest reflect the quality and objectivity of the individuals conducting the inquiry. Interviewing the respondent, whistleblower and witnesses and utilizing various data sources provide the information needed to make the decision whether an investigation is warranted. The analysis provides the interpretation of the data and logically links the data to the decision. Comments by the respondent test the decision.
All criteria need not be met in all inquiries. An inquiry into plagiarism may not require interviews of the whistleblower or witnesses and may not require expertise in the discipline. A respondent may choose not to comment on the report because no recommendation for an investigation was made. No one criterion constitutes a necessary and sufficient condition upon which a decision may be made. The analysis of the evidence probably comes closest to fulfilling that requirement.
The percentage of reports containing each criterion is liberally estimated for it includes reports that explicitly addressed the criterion as well as reports that partially addressed the criterion. The descriptions "partially addressed" included information that was either redacted, or cited as an appendix or attachment to the report but not submitted with the report. The absence of information in the report about a criterion does not necessarily mean that the institution did not take the appropriate action. It may have been documented elsewhere or not documented at all.
Table 2:Percentage of reports containing some information on each criterion for evaluation whether an investigation was warranted.
| Criterion | Percentage |
|---|---|
| Appropriate expertise | 57 |
| Addressed conflicts of interest | 29 |
| Interview respondent | 81 |
| Interview whistleblower | 62 |
| Interview witnesses | 47 |
| Cited sources of evidence | 81 |
| Presented analysis of evidence | 48 |
| Provided report to respondent for comment | 29 |
| Attached respondent's comments to report | 24 |
Appropriate Expertise
Only 3 (14%) of the 21 reports provided enough information to indicate that appropriate expertise was available during the inquiry. Appropriate expertise may also have been utilized in nine (43%) inquiries, but redaction of the disciplines of the committee members made a definitive determination impossible. Nine (43%) other reports did not contain any indication that appropriate experts served on the inquiry panel.
To determine whether inquiry committee members had appropriate expertise, it was necessary for the report to contain information on the discipline and research area of the respondent. Five (23%) reports identified the respondent's area of research; two of these reports demonstrated a match with the committee membership. Four (19%) reports did not state this information, and 12 (57%) did not contain sufficient information to identify the discipline of the respondent.
Conflicts of Interest
Six (29%) of the reports discussed possible conflicts of interest on the part of inquiry committee members. These reports either identified a conflict and corrected it by substituting an appropriate committee member, or stated that there were no conflicts. Fifteen (71%) reports did not address conflicts of interest or whether steps were taken to protect against it.
Interview Respondent
Respondent interviews were reported most often. Thirteen (62%) reports demonstrated that the respondents were interviewed. However, three interviews were conducted by the respondent providing written responses to inquiry committee questions. Interviews may have been reported in four (19%) other reports but redactions made a definitive determination impossible. The remaining four (19%) reports did not indicate the respondent was interviewed by any means.
Interview Whistleblower
Nine (43%) reports indicated that the whistleblower was interviewed; interviews may have been reported in four (19%) other reports but redactions made a definitive determination impossible. Eight (38%) reports did not indicate the whistleblower was interviewed.
Interview Witnesses
Witness interviews were represented the least. Seven (33%) reports indicated that witnesses were interviewed; interviews may have been reported in three (14%) reports but redactions made a definitive determination impossible. Eleven (52%) reports contained no information on witness interviews. It is possible that there were no witnesses to interview in some cases.
Cited Sources of Evidence
Seventeen (81%) reports provided a list of evidence sources consulted including lab notebooks, interview summaries, grant applications, pre-published manuscripts, published papers, correspondence, curriculum vitae, slides, computer files, purchase orders, doctoral theses, personnel file documents, and fellowship applications. However, these reports did not describe the evidence provided by these sources in detail. Four (19%) reports provided no specific information on the sources of evidence reviewed.
Analysis of Evidence
The heart of the inquiry report is a reasoned analysis that links the detailed evidence to the conclusion that an investigation is unwarranted. Ten (48%) reports provided such an analysis. Eleven (52%) reports did not.
Provided Inquiry Report to Respondent
Five (24%) reports indicated a copy of the report was provided to the respondent for comment. Sixteen (76%) reports did not include information indicating whether the respondent was afforded the opportunity to comment. It appears that many institutions may not provide the respondent with the report for comment because an investigation is not recommended. However, reports frequently discuss other problems uncovered by the inquiry that could affect the reputation and position of the respondent. In some cases, the inquiry report may be given to the respondent by the institutional official reviewing the inquiry committee report, and therefore, such action is not noted in the report.
Respondent's Comments Attached to Report
Five (24%) reports included the respondent's comments. Sixteen (76%) reports gave no indication whether the respondent had commented on the report. The regulation does not require institutions to make the comments from the respondent a part of the record. However, those comments are useful in evaluating whether the inquiry was objective, thorough, and competent.
The number of criteria covered in a report ranged from none to eight; none covered nine criteria. (See Table 3.) The number of combinations produced by nine criteria is enormous, so it is difficult to stipulate what combination of criteria must be covered in a report, particularly when the qualitative implementation of each criterion is paramount. Nevertheless, reports limited to four or fewer criteria are not likely to provide sufficient support for a decision on whether to investigate. Reports covering five criteria are probably marginal while those including six or more criteria are most likely sufficient.
Using the scale cited in the above paragraph, the seven reports (33%) covering four or fewer criteria were deficient; six (29%) reports containing five criteria were marginal, and eight (38%) reports addressing six or more criteria were sufficient especially if they included an analysis of the evidence.
Table 3: Number of reports by number of criteria evident in a report for evaluating whether an investigation was warranted.
| Criteria | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
| Reports | 2 | 1 | 2 | 1 | 1 | 6 | 6 | 1 | 1 | 0 |
COMPLIANCE WITH REGULATION
Besides monitoring the conduct of inquiries to determine whether they are thorough, objective and competent, ORI also is responsible for ensuring institutional compliance with the regulation. Therefore, the inquiry reports were analyzed from a compliance perspective to determine whether they contained information on the nine regulatory provisions that must be implemented in an inquiry. (See Table 4.) The first four provisions listed in the table are directly related to deciding whether an investigation is warranted. The five other provisions are concerned with the length of the inquiry and the protection of the respondent, whistleblower, and Federal funds.
As expected, the reports provided information on the provisions related to the conduct of an inquiry, but contained no information on the required efforts to protect respondents and whistleblowers or the imposition of interim administrative actions to protect Federal funds. As with the provision on exceeding the 60-day standard, the provisions related to the respondent, whistleblower, and interim administrative actions may not be implemented in all inquiries because the situation may not require their implementation, but such situations could be noted in the report. The compliance information may be missing because the regulation stipulates only that an inquiry report must contain the evidence reviewed, summarize relevant interviews and include the conclusions of the inquiry.
Table 4: Percentage of reports containing information on each regulatory provision.
| Provision | Percentage |
|---|---|
| Appropriate expertise | 57 |
| Address conflict of interest | 29 |
| Permit respondent to comment on allegation | 81 |
| Provided report to respondent for comment | 29 |
| Maintain confidentiality | 19 |
| Reasons for exceeding 60-days | 0 |
| Restoring reputation of exonerated respondent | 0 |
| Protecting position of whistleblower | 0 |
| Interim administrative actions | 0 |
Maintain Confidentiality
Four (19%) reports contained information on the steps taken to maintain confidentiality. Seventeen (81%) reports did not.
Reasons for Exceeding 60 Days
This requirement clearly applied to only the two (10%) inquiries that lasted at least 2 years. The reports on these inquiries did not contain any explanation for the time extension. Twelve (57%) inquiries were completed within the 60-day standard set by the regulation.6 The processing time for the remaining seven (33%) inquiries could not be determined.
The results presented in Table 5 clearly show that the inquiry reports did not contain sufficient information to determine whether the inquiries complied with the regulation.
Table 5: Number of reports by number of regulatory provisions addressed in the report.
| Provisions | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
| Reports | 4 | 6 | 5 | 5 | 1 | 0 | 0 | 0 | 0 |
REPORT LENGTH
Although the length of a report is not indicative of the quality of the information contained therein, the study indicates that it is very difficult to adequately present the detailed information required to demonstrate that the decision not to proceed to an investigation was based on a thorough, objective and competent inquiry in less than a 5-page report. Fifty-seven percent of the reports analyzed in the study were less than 5 pages in length while 24 percent were 11 pages or more. (See Table 6.)
Table 6: Number of inquiry reports by number of pages in a report.
| Number of Pages | Reports |
|---|---|
| 1-2 | 7 |
| 3-4 | 5 |
| 5-6 | 2 |
| 7-8 | 1 |
| 9-10 | 1 |
| 11 or more | 5 |
| Total | 21 |
CONCLUSION
This study demonstrated that more than half of the reports on inquiries that were not reported to ORI were significantly deficient. Fifty-seven percent did not contain information establishing PHS jurisdiction. Thirty-three percent contained information on no more than four of the nine criteria used to determine whether an investigation was warranted and another 28 percent were marginal, covering only five criteria. Seventy-one percent provided information on only three or fewer criteria for determining compliance with the regulation. And finally, 57 percent of the reports did not contain the detailed information required to justify the decision that an investigation is unwarranted.
These findings suggest that more technical assistance should be provided in the following areas:
-
- Establishing PHS jurisdiction.
- Interpreting the PHS definition of scientific misconduct, especially plagiarism and "other practices."
- Conducting a thorough, objective, and competent inquiry.
- Preparing inquiry reports in a manner that demonstrates that a decision not to proceed to an investigation is warranted.
- Reporting information that supports compliance with the regulation.
Attachments:
-
- Letter requesting participation in the study.
- Coding form for study.
___________________________________________
1 42 C.F.R. Part 50, Subpart A, § 50.103 (d)(6).
2Institutions are required to report the total number of inquires they conduct each year and indicate whether each inquiry resulted in an investigation.
3One inquiry report was not included in the study because the inquiry was terminated shortly after it began upon the death of the respondent.
442 C.F.R. Part 50, Subpart A.
5ORI provides its working definition of plagiarism in ORI Newsletter 1994; 3(1):3.
642 C.F.R. Part 50, Subpart A, § 50.103 (d)(1).
November 7, 1996
Dear
The Office of Research Integrity (ORI) is conducting a retrospective study of institutional scientific misconduct inquiries that did not proceed to an investigation to gain insight into the implementation of the Federal regulation (42 CFR Part 50, Subpart A) by institutions and identify what technical assistance and educational efforts, if any, ORI may be able to offer to institutions with respect to the conduct of inquiries and investigations.
We are requesting that you submit the final report and documentation for the (inquiry or # inquiries) that you reported in the 1994 or 1995 Annual Report of Possible Research Misconduct for which no investigation was reported. The study will compare the inquiry process actually employed by institutions with the provisions of the Federal regulation.
In submitting the requested materials, the names of all institutions and individuals should be redacted to protect the confidentiality of the proceedings and the privacy of the individuals involved. The material should be submitted in a plain envelope with no return address. A letter should be sent to ORI under separate cover indicating your response to this request. If you prefer, ORI will do the redaction.
Study results will be reported as aggregated, non-identifiable data in the ORI report which will be sent to each participating institution. The report also will be generally available from ORI upon request, the results will be published in the ORI Newsletter and the ORI Annual Report.
We would appreciated receiving the materials requested above by December 15, 1996. Your cooperation with this study is greatly appreciated. If you have any questions regarding the proposed study, please call me at (301) 443-5300.
Sincerely,
Lawrence J. Rhoades, Ph.D.
Director
Division of Policy and Education
INQUIRY QUESTIONNAIRE
After reading the Inquiry reports, indicate with an "x" whether the information exists that complies with the Federal regulation.
Yes indicates that the report clearly addresses the provision cited for the Federal regulation.
No indicates that the report does not address the provision cited for the Federal regulation.
P/A indicates that the report eludes to addressing the provision cited for the Federal regulation but the information is not clear enough to be certain that the provision is addressed.
N/A indicates that the information is not applicable to this inquiry report.
Any unique features not highlighted in the Federal regulation may be added in the comment section.
Is the allegation of scientific misconduct identified in the inquiry report?
Yes _____ No _____
Is the PHS funding support identified in the inquiry report?
Yes _____ No _____
Does the inquiry report identify the respondent's discipline or area of research?
Yes_____ No ______ P/A _____ N/A ______
Comment:
Does the inquiry report describe a selection of necessary and appropriate expertise for inquiries? [§50.103 (d)(8)]
Yes_____ No ______ P/A _____ N/A ______
Comment:
Does the report indicate whether the affected individual(s) were afforded confidential treatment to the maximum extent possible during the inquiry? [§50.103 (d) (2,3)]
Yes_____ No ______ P/A _____ N/A ______
Comment:
Was the respondent interviewed?
Yes_____ No ______ P/A _____ N/A ______
Comment:
Was the complainant interviewed?
Yes_____ No ______ P/A _____ N/A ______
Comment:
Were witnesses interviewed?
Yes_____ No ______ P/A _____ N/A ______
Comment:
Does the inquiry report show that the [respondent(s)] were afforded an opportunity to comment on allegations and findings of the inquiry. [§50.103 (d)(1,3) and §50.104 (a)(2)]
Yes_____ No ______ P/A _____ N/A ______
Comment:
Are the respondent's comments included with the inquiry report?
Yes_____ No ______ P/A _____ N/A ______
Comment:
Was the inquiry, including the report, completed within 60 calendar days of its initiation? [§50.103 (d)(1)] (Date of inquiry begins at the date of the 1st meeting of committee)
Yes_____ No ______ P/A _____ N/A _______
Comment:
If the inquiry and the completion of the inquiry report were not completed within 60 days, is there documentation of reasons for extending the inquiry beyond 60 calendar days? [§50.103 (d)(1)]
Yes_____ No ______ P/A _____ N/A _____
Comment:
Did the inquiry report include evidence reviewed for making a determination not to go forward? [§50.103 (d)(1)]
Yes_____ No ______ P/A _____ N/A _____
Comment:
If "yes" is answered above, check what information was noted in the inquiry report: [§50.103 (d)(1)]
Interview summaries ____
Purchase Orders ______
Lab notebooks _____
Animal Use Records ______
Computer files _____
Other Records (indicate) ______
___________________________
Does the inquiry report include the conclusions of the inquiry report? [§50.104 (d)(1)]
Yes_____ No ______ P/A _____ N/A ______
Comment:
Indicate with an "x" whether the inquiry report was conducted by 1 person, or committee:
______ One person
______ Committee
______ Not addressed
Does the inquiry report mention the prevention of real or apparent conflicts of interest? [§50.103 (d)(9)]
Yes_____ No ______ P/A _____ N/A ______
Comment:
Did the ORI need to be notified about an immediate health hazards, need to protect Federal funds or equipment and individuals affected by the inquiry, and that the alleged incident will probably be publicly reported. If reasonable indication of possible criminal violations was found, was ORI notified within 24 hours.? [§50.104 (b)(1-5)]
Yes_____ No ______ P/A_____ N/A ______
Comment:
Does the report indicate whether appropriate interim administrative actions had to be taken to protect Federal funds and ensure that the purposes of the Federal financial assistance were being carried out? [§50.103 (d)(11)]
Yes_____ No ______ P/A_____ N/A ______
Comment:
Did the institution terminate the inquiry for any reason without completing all relevant requirements under [§50.103 (d)]?
Yes_____ No ______ P/A_____ N/A ______
Comment:
If "yes" is answered above, does the inquiry report, or attachment give a description of the reasons for such termination? [§50.104 (a)(3)]
Yes_____ No ______ P/A_____ N/A ______
Comment:
Does the inquiry report include information regarding the necessity to undertake diligent efforts, as appropriate, to restore the reputations of persons alleged to have engaged in misconduct when allegations are not confirmed? [§50.103 (d)(13)]
Yes_____ No ______ P/A _____ N/A _______
Comment:
Does the inquiry report include information regarding the necessity to undertake diligent efforts to protect the positions and reputations of those persons who, in good faith, make allegations of scientific misconduct [§50.103 (d)(13)]
Yes_____ No ______ P/A _____ N/A ______
Comment:
Check with an "x" the total number of pages in the inquiry report.
1-2 _____ 5-6 _____ > 10_____
3-4 _____ 7-8 _____
4-5 _____ 9-10 ____