Publications
Publications
- 2023
- HBS Working Paper Series
Chatbots and Mental Health: Insights into the Safety of Generative AI
By: Julian De Freitas, Ahmet Kaan Uğuralp, Zeliha Uğuralp and Stefano Puntoni
Abstract
Chatbots are now able to engage in sophisticated conversations with consumers. Due to the ‘black box’ nature of the algorithms, it is impossible to predict in advance how these conversations will unfold. Behavioral research provides little insight into potential safety issues emerging from the current rapid deployment of this technology at scale. We begin to address this urgent question by focusing on the context of mental health and “companion AI”: applications designed to provide consumers with synthetic interaction partners. A Pilot Study reports an extensive performance test of several commercially available companion AIs. Studies 1 and 2 present field evidence: actual consumer interactions with two different companion AIs. Study 3 is an experiment testing consumer reaction to risky and unhelpful chatbot responses. The findings show that (1) companion AIs are often unable to recognize, and respond appropriately to, signs of distress; (2) mental health crises are apparent in a non-negligible minority of conversations with users; and (3) consumers display negative reactions to unhelpful and risky chatbot responses, highlighting emerging reputational risks for generative AI companies.
Keywords
Autonomy; Artificial Intelligence; Chatbots; New Technology; Brand Crises; Ethics; Mental Health; AI and Machine Learning; Well-being; Health; Applications and Software
Citation
De Freitas, Julian, Ahmet Kaan Uğuralp, Zeliha Uğuralp, and Stefano Puntoni. "Chatbots and Mental Health: Insights into the Safety of Generative AI." Harvard Business School Working Paper, No. 23-011, August 2022. (Revised July 2023.)