Publications
Publications
- November 9, 2019
- BMJ: British Medical Journal
Effect of Revealing Authors' Conflicts of Interests in Peer Review: Randomized Controlled Trial
By: Leslie K. John, George Loewenstein, Andrew Marder and Michael Callaham
Abstract
Objective: To assess the impact of disclosing authors’ conflict of interest declarations to peer reviewers at a medical journal.
Design: Randomised controlled trial.
Setting: The study was conducted within the manuscript review process at the Annals of Emergency Medicine.
Participants: Reviewers (n=885 reviewers) who reviewed manuscripts submitted between June 2, 2014 and January 23, 2018 inclusive (n=1480 manuscripts).
Intervention: Reviewers were randomised to either receive (treatment) or to not receive (control) authors’ full ICMJE-format conflict of interest disclosures before performing their review. Reviewers rated the manuscripts as usual on eight quality ratings and were then surveyed to obtain “counterfactual scores”—i.e., the scores they believed they would have given had they been assigned to the opposite arm—as well as attitudes toward conflicts of interest. .
Main outcome measures: The primary outcome measure was the overall quality score that reviewers assigned to the manuscript upon submitting their review (1 to 5 scale). The trial had 99% power to detect a .4 point difference, determined previously by editors to be the minimal significance threshold for a difference in score that would influence their editorial decision. Secondary outcomes looked at the scores the reviewers submitted for the seven more specific quality ratings as well as counterfactual scores elicited in the follow-up survey.
Results: Providing authors’ conflict of interest disclosures did not affect reviewers’ ratings of manuscript quality (Mcontrol=2.70 out of 5, SD=1.11; Mtreatment=2.74 out of 5, SD=1.13; mean difference=0.04, 95% CI=-0.05 to 0.14), even for manuscripts with disclosed conflicts (Mcontrol=2.85 out of 5, SD=1.12; Mtreatment=2.96 out of 5, SD=1.16; mean difference=0.11, 95% CI=-0.05 to 0.26). Similarly, there was no effect of the treatment on any of the other seven quality ratings the reviewers assigned. Reviewers acknowledged conflicts of interest as an important issue and believed they could correct for them when disclosed. Yet, their counterfactual scores did not differ from actual scores (Mactual=2.69, Mcounterfactual=2.67; difference in means=0.02, 95% CI=-0.02 to -0.01). When conflicts were reported, a comparison of different source types (e.g., government, for-profit corporation, etc.) found no difference in impact.
Conclusions: Current ethical standards require conflict of interest disclosure for all scientific reports. As currently implemented, this practice demonstrated no impact on any quality ratings of real manuscripts being evaluated for publication by real peer reviewers.
Design: Randomised controlled trial.
Setting: The study was conducted within the manuscript review process at the Annals of Emergency Medicine.
Participants: Reviewers (n=885 reviewers) who reviewed manuscripts submitted between June 2, 2014 and January 23, 2018 inclusive (n=1480 manuscripts).
Intervention: Reviewers were randomised to either receive (treatment) or to not receive (control) authors’ full ICMJE-format conflict of interest disclosures before performing their review. Reviewers rated the manuscripts as usual on eight quality ratings and were then surveyed to obtain “counterfactual scores”—i.e., the scores they believed they would have given had they been assigned to the opposite arm—as well as attitudes toward conflicts of interest. .
Main outcome measures: The primary outcome measure was the overall quality score that reviewers assigned to the manuscript upon submitting their review (1 to 5 scale). The trial had 99% power to detect a .4 point difference, determined previously by editors to be the minimal significance threshold for a difference in score that would influence their editorial decision. Secondary outcomes looked at the scores the reviewers submitted for the seven more specific quality ratings as well as counterfactual scores elicited in the follow-up survey.
Results: Providing authors’ conflict of interest disclosures did not affect reviewers’ ratings of manuscript quality (Mcontrol=2.70 out of 5, SD=1.11; Mtreatment=2.74 out of 5, SD=1.13; mean difference=0.04, 95% CI=-0.05 to 0.14), even for manuscripts with disclosed conflicts (Mcontrol=2.85 out of 5, SD=1.12; Mtreatment=2.96 out of 5, SD=1.16; mean difference=0.11, 95% CI=-0.05 to 0.26). Similarly, there was no effect of the treatment on any of the other seven quality ratings the reviewers assigned. Reviewers acknowledged conflicts of interest as an important issue and believed they could correct for them when disclosed. Yet, their counterfactual scores did not differ from actual scores (Mactual=2.69, Mcounterfactual=2.67; difference in means=0.02, 95% CI=-0.02 to -0.01). When conflicts were reported, a comparison of different source types (e.g., government, for-profit corporation, etc.) found no difference in impact.
Conclusions: Current ethical standards require conflict of interest disclosure for all scientific reports. As currently implemented, this practice demonstrated no impact on any quality ratings of real manuscripts being evaluated for publication by real peer reviewers.
Keywords
Conflicts Of Interest; Peer Review; Randomized Controlled Trial; Scientific Publication; Conflict Of Interests; Journals And Magazines; Science
Citation
John, Leslie K., George Loewenstein, Andrew Marder, and Michael Callaham. "Effect of Revealing Authors' Conflicts of Interests in Peer Review: Randomized Controlled Trial." BMJ: British Medical Journal 367, no. 8221 (November 9, 2019).
About The Author
