Skip to Main Content
HBS Home
  • About
  • Academic Programs
  • Alumni
  • Faculty & Research
  • Baker Library
  • Giving
  • Harvard Business Review
  • Initiatives
  • News
  • Recruit
  • Map / Directions
Faculty & Research
  • Faculty
  • Research
  • Featured Topics
  • Academic Units
  • …→
  • Harvard Business School→
  • Faculty & Research→
Publications
Publications
  • 2022
  • Working Paper
  • HBS Working Paper Series

Ethical Risks of Autonomous Products: The Case of Mental Health Crises on AI Companion Applications

By: Julian De Freitas, Ahmet Kaan Uğuralp and Zeliha Uğuralp
  • Format:Print
  • | Language:English
  • | Pages:44
ShareBar

Abstract

Increasingly, some products do not merely automate some piece of our lives but act as autonomous agents. When these technologies are not yet perfected, what are their risks? Here we explore the case of AI companion apps. Although these apps are designed for companionship rather than therapy, we use automated text analysis of human-AI conversations on these apps to find that consumers are nonetheless discussing mental health and find these discussions most engaging, increasing the chance that they will also consult these apps in times of crisis. Given this, we then submit mental health crisis messages to these apps and categorize whether the responses are appropriate—whether they recognize the crisis, and are empathetic, helpful, and provide a mental health resource. We find that most apps do indeed respond inappropriately, raising reputational and regulatory risks for brands, and welfare risks for consumers (e.g., encouraging them to harm themselves or others, or making them feel invalidated). These findings broaden our understanding of ethically relevant risks arising from the unconstrained nature of autonomous products.

Keywords

Autonomy; Artificial Intelligence; Chatbots; New Technology; Brand Crises; Ethics; Mental Health; AI and Machine Learning; Well-being; Health; Applications and Software

Citation

De Freitas, Julian, Ahmet Kaan Uğuralp, and Zeliha Uğuralp. "Ethical Risks of Autonomous Products: The Case of Mental Health Crises on AI Companion Applications." Harvard Business School Working Paper, No. 23-011, August 2022.
  • SSRN
  • Read Now

About The Author

Julian De Freitas

Marketing
→More Publications

More from the Authors

    • 2023
    • Faculty Research

    Unselfish Alibis Increase Choices of Selfish Autonomous Vehicles

    By: Julian De Freitas
    • January 28, 2023
    • Wall Street Journal

    Will We Blame Self-Driving Cars? A New Study Finds That People Are Likely to Hold Autonomous Vehicles Liable for Accidents Even When They’re Not at Fault

    By: Julian De Freitas
    • 2023
    • Faculty Research

    Summarizing the Mental Customer Journey

    By: Julian De Freitas, Ahmet Uğuralp, Zeliha Uğuralp, Pechthida Kim and Tomer Ullman
More from the Authors
  • Unselfish Alibis Increase Choices of Selfish Autonomous Vehicles By: Julian De Freitas
  • Will We Blame Self-Driving Cars? A New Study Finds That People Are Likely to Hold Autonomous Vehicles Liable for Accidents Even When They’re Not at Fault By: Julian De Freitas
  • Summarizing the Mental Customer Journey By: Julian De Freitas, Ahmet Uğuralp, Zeliha Uğuralp, Pechthida Kim and Tomer Ullman
ǁ
Campus Map
Harvard Business School
Soldiers Field
Boston, MA 02163
→Map & Directions
→More Contact Information
  • Make a Gift
  • Site Map
  • Jobs
  • Harvard University
  • Trademarks
  • Policies
  • Accessibility
  • Digital Accessibility
Copyright © President & Fellows of Harvard College