Effect of Revealing Authors' Conflicts of Interests in Peer Review: Randomized Controlled Trial

Leslie K. John; George  Loewenstein; Andrew  Marder; Michael  Callaham

Article | BMJ: British Medical Journal | November 9, 2019

Effect of Revealing Authors' Conflicts of Interests in Peer Review: Randomized Controlled Trial

by Leslie K. John, George Loewenstein, Andrew Marder and Michael Callaham

Abstract

Objective: To assess the impact of disclosing authors’ conflict of interest declarations to peer reviewers at a medical journal.
Design: Randomised controlled trial.

Setting: The study was conducted within the manuscript review process at the Annals of Emergency Medicine.

Participants: Reviewers (n=885 reviewers) who reviewed manuscripts submitted between June 2, 2014 and January 23, 2018 inclusive (n=1480 manuscripts).

Intervention: Reviewers were randomised to either receive (treatment) or to not receive (control) authors’ full ICMJE-format conflict of interest disclosures before performing their review. Reviewers rated the manuscripts as usual on eight quality ratings and were then surveyed to obtain “counterfactual scores”—i.e., the scores they believed they would have given had they been assigned to the opposite arm—as well as attitudes toward conflicts of interest. .

Main outcome measures: The primary outcome measure was the overall quality score that reviewers assigned to the manuscript upon submitting their review (1 to 5 scale). The trial had 99% power to detect a .4 point difference, determined previously by editors to be the minimal significance threshold for a difference in score that would influence their editorial decision. Secondary outcomes looked at the scores the reviewers submitted for the seven more specific quality ratings as well as counterfactual scores elicited in the follow-up survey.

Results: Providing authors’ conflict of interest disclosures did not affect reviewers’ ratings of manuscript quality (Mcontrol=2.70 out of 5, SD=1.11; Mtreatment=2.74 out of 5, SD=1.13; mean difference=0.04, 95% CI=-0.05 to 0.14), even for manuscripts with disclosed conflicts (Mcontrol=2.85 out of 5, SD=1.12; Mtreatment=2.96 out of 5, SD=1.16; mean difference=0.11, 95% CI=-0.05 to 0.26). Similarly, there was no effect of the treatment on any of the other seven quality ratings the reviewers assigned. Reviewers acknowledged conflicts of interest as an important issue and believed they could correct for them when disclosed. Yet, their counterfactual scores did not differ from actual scores (Mactual=2.69, Mcounterfactual=2.67; difference in means=0.02, 95% CI=-0.02 to -0.01). When conflicts were reported, a comparison of different source types (e.g., government, for-profit corporation, etc.) found no difference in impact.

Conclusions: Current ethical standards require conflict of interest disclosure for all scientific reports. As currently implemented, this practice demonstrated no impact on any quality ratings of real manuscripts being evaluated for publication by real peer reviewers.

Keywords: conflicts of interest; peer review; randomized controlled trial; scientific publication; Conflict of Interests; Journals and Magazines; Science;

Format: Electronic

Find at Harvard Read Now

Citation:

John, Leslie K., George Loewenstein, Andrew Marder, and Michael Callaham. "Effect of Revealing Authors' Conflicts of Interests in Peer Review: Randomized Controlled Trial." BMJ: British Medical Journal 367, no. 8221 (November 9, 2019).

Email mking@hbs.edu for a courtesy copy.

In August 2017, Commonwealth Bank of Australia was looking for ways to differentiate itself from competing banks and was also trying to improve the financial well-being of its customers. One domain where this was particularly relevant was in its bank-issued credit card business, where customers routinely selected cards that although profitable for the bank could be a poor fit for customers’ needs—leading to low satisfaction scores, cancellations, and occasionally, financial distress. To that end, the company’s Behavioral Economics team had developed a provocative experiment dubbed “The Good and the Bad.” Rather than just presenting the strengths of its various credit card offerings, they proposed also promoting each credit card’s less-obvious drawbacks. Being transparent with customers might help them make better choices, but would those choices come at the expense of bank performance? Should a company choose to be in the sales prevention business?

Email mking@hbs.edu for a courtesy copy.

This Teaching Note explains the theory of the case and teaching plan for the case: Commonwealth Bank of Australia: Unbanklike Experimentation (619-018). In August 2017, Commonwealth Bank of Australia was looking for ways to differentiate itself from competing banks and was also trying to improve the financial well-being of its customers. One domain where this was particularly relevant was in its bank-issued credit card business, where customers routinely selected cards that although profitable for the bank could be a poor fit for customers’ needs—leading to low satisfaction scores, cancellations, and occasionally, financial distress. To that end, the company’s Behavioral Economics team had developed a provocative experiment dubbed “The Good and the Bad.” Rather than just presenting the strengths of its various credit card offerings, they proposed also promoting each credit card’s less-obvious drawbacks. Being transparent with customers might help them make better choices, but would those choices come at the expense of bank performance? Should a company choose to be in the sales prevention business?

Case | HBS Case Collection | December 2018 (Revised February 2020)

Fishbowl: Scaling Up

Leslie K. John

Fishbowl is a social media app that allows professionals to connect with other relevant professionals both within their company and across industry. Unlike many other social media apps, on which users typically present idealized portraits of themselves, on Fishbowl, people get real. Fishbowl prides itself in being a "safe space" that allows users to feel comfortable interacting with candor - whether to ask difficult questions in order to give and get advice, or just to vent or crack jokes. A key part of the user experience is the ability to post anonymously. But to ensure relevance of posts, when a user signs up, Fishbowl verifies their identity by requiring them to provide their full name, employer email address, LinkedIn account and contact list. Fishbowl has several hundred thousand users and is now looking for ways to monetize the platform. As such, founders Loren Appin and Matt Sunbulli face a mission-critical decision: should they integrate employers into the platform? Although formally integrating employers would provide a much-needed revenue stream, at the same time Appin and Sunbulli worry that doing so could destroy the user experience. Is formalizing employer relationships antithetical to the safe space they have created?

Keywords: communication technologies; customer value; value chain; Interpersonal Communication; Talent and Talent Management; Customer Value and Value Chain; Entrepreneurship; Business Model; Growth and Development Strategy; Marketing Strategy; Advertising; Product Marketing; Two-Sided Platforms; Consumer Behavior; Network Effects; Emotions; Motivation and Incentives; Trust; Software; Technology Adoption; Technology Platform; Communications Industry; Employment Industry; Media and Broadcasting Industry; Technology Industry; Telecommunications Industry; United States;

Citation:

John, Leslie K. "Fishbowl: Scaling Up." Harvard Business School Case 919-013, December 2018. (Revised February 2020.) (Email mking@hbs.edu for a courtesy copy.) View Details

Cite View Details Educators Purchase

Faculty & Research

Effect of Revealing Authors' Conflicts of Interests in Peer Review: Randomized Controlled Trial

Abstract

Citation:

About the Author

More from the Author