"I tell him everything that I do": An investigation of privacy and safety implications of AI companion usage

The advances of generative AI and in particular large language models (LLM) resulted in the proliferation of AI chatbots designed for a variety of functions. One example of such chatbots are the so-called AI companion apps that allow creating an anthropomorphised character one can interact with. Indeed, AI companions become an increasingly common part of people’s daily lives resulting in increased risks of adverse privacy and safety consequences. In this work we investigate the experiences of users of the Replika chatbot, an AI companion app that is advertised as “the AI companion who cares”. We analyse 111 Reddit posts of Replika users, focusing on data shared with the app as well as harms users experience from interacting with the app. Our analysis shows that Replika is commonly seen as a simulation of human relationships, which results in users being attached to their chatbot in a similar way they would be attached to their romantic partner or a close friend. Such an attachment leads to significant amounts of sensitive data shared with the Replika app such as details about one’s personal life, mental health issues, or sexual preferences. On the other hand, unexpected changes in Replika’s workings, e.g., due to new restrictions or bugs introduced by software updates, elicit strong reactions among its users who report harms akin to feeling betrayed or abandoned by a real-life companion. Our research concludes the need for further investigation of relationships to AI companions and of possible ways to mitigate these privacy and safety risks.