Publications
Publications
- 2022
- HBS Working Paper Series
Ethical Risks of Autonomous Products: The Case of Mental Health Crises on AI Companion Applications
By: Julian De Freitas, Ahmet Kaan Uğuralp and Zeliha Uğuralp
Abstract
Increasingly, some products do not merely automate some piece of our lives but act as
autonomous agents. When these technologies are not yet perfected, what are their risks? Here
we explore the case of AI companion apps. Although these apps are designed for
companionship rather than therapy, we use automated text analysis of human-AI
conversations on these apps to find that consumers are nonetheless discussing mental health
and find these discussions most engaging, increasing the chance that they will also consult
these apps in times of crisis. Given this, we then submit mental health crisis messages to these
apps and categorize whether the responses are appropriate—whether they recognize the crisis,
and are empathetic, helpful, and provide a mental health resource. We find that most apps do
indeed respond inappropriately, raising reputational and regulatory risks for brands, and
welfare risks for consumers (e.g., encouraging them to harm themselves or others, or making
them feel invalidated). These findings broaden our understanding of ethically relevant risks
arising from the unconstrained nature of autonomous products.
Keywords
Autonomy; Artificial Intelligence; Chatbots; New Technology; Brand Crises; Ethics; Mental Health; AI and Machine Learning; Well-being; Health; Applications and Software
Citation
De Freitas, Julian, Ahmet Kaan Uğuralp, and Zeliha Uğuralp. "Ethical Risks of Autonomous Products: The Case of Mental Health Crises on AI Companion Applications." Harvard Business School Working Paper, No. 23-011, August 2022.