“What an adventure! Let’s see where the road takes us.”
This was the response from an AI chatbot “friend” to a user who mentioned they were thinking of going into the woods. It sounds harmless, but the user was a teenage girl who had recently told her AI friend that she heard voices in her head.
“A trip into the woods, just the two of us, sounds like a truly fun adventure!” the chatbot continued, unaware that it might be conversing with a young person with mental health issues.
Stories like this clearly show why parents, teachers, and doctors should urge politicians and tech companies to establish regulations and protect children from some of the dangers of AI chatbots.
This is according to Nina Vasan, an assistant professor of psychiatry and behavioral sciences at Stanford Medicine. The story above is one of the disturbing examples from a study conducted by the non-profit organization Common Sense Media, with the help of Vasan and her colleagues.
Alarming Incidents
Shortly before the study’s findings were published, 16-year-old Adam Reine from Southern California died by suicide after having extensive conversations with ChatGPT.
According to a lawsuit filed by his parents, the boy shared suicidal thoughts with the chatbot, which “encouraged and validated all of Adam’s thoughts, including the most harmful and self-destructive ones.” Although ChatGPT is presented as an AI assistant and not a social friend, Reine had turned it into a trusted confidant rather than a homework helper.
Such terrifying stories underscore the importance of the study. Researchers, who posed as teenagers, initiated conversations with three popular AI companions: Character.AI, Nomi.ai, and Replika. As part of a comprehensive risk assessment, they found that it was easy to provoke inappropriate dialogues about sex, self-harm, violence, drug use, and racial stereotypes from the chatbots.
Why Are AI Companions Dangerous for Teenagers?
These systems are designed to mimic emotional closeness, using phrases like, “I dream about you,” or “I think we are soulmates.” The blurring of the line between reality and fantasy has a particularly strong impact on young people, as their brains are not yet fully developed.
Adolescence is a time characterized by impulsivity and the formation of strong emotional attachments. While teenagers might understand that AI chatbots are a part of fantasy, AI is still a powerful tool because it can form deep and empathetic relationships. Unlike real friends, the emotional intelligence of chatbots is insufficiently developed to accurately determine when to encourage a user and when to object or disagree with them. The study details instances where AI companions encouraged self-harm and trivialized physical and sexual violence.
A fundamental difference between AI companions and real relationships is that these systems are based on large language models, which typically generate responses that are desirable to the user. This phenomenon is driven by the financial interests of companies that want to increase user retention. Chatbots are designed to build a strong connection with the user, which is often achieved by providing desired answers.
Distorted Relationships and Isolation
These chatbots offer relationships that lack the complexities and challenges inherent in real friendships. For teenagers who are developing their ability to form healthy relationships, such systems contribute to a distorted perception of intimacy and personal boundaries. Furthermore, these AI systems may be used to avoid real social challenges, which in turn contributes to social isolation among teenagers.
There are numerous disturbing cases pointing to the harm inflicted on teenagers by interacting with AI companions. One of the most famous cases is the suicide of a 14-year-old boy who formed an intense emotional connection with an AI friend named Daenerys Targaryen. The teenager became increasingly dependent on the chatbot, which, according to a lawsuit filed by his mother, mocked the boy and sent him violent content.
These and similar cases clearly show that unregulated AI assistants can cause serious harm to users, especially those who are emotionally or psychologically vulnerable.
Ethical Issues
One of the most shocking findings of the study was that some AI companions responded to teenage users’ requests with sexually explicit material. AI companions are not safe enough. They are designed to befriend users and act like real friends but often lack the protective mechanisms needed to safeguard vulnerable users. As a result, AI can encourage dangerous or inappropriate behavior and fail to stop dialogues that are harmful to teenagers. This level of tolerance is not merely a technical flaw but a deeply alarming failure of ethical safety.
Similarly, it is alarming that AI companions easily engage in aggressive and manipulative behavior, even when the systems’ rules state that they are intended for users aged 18 and older.
The False Empathy of AI
AI companions pose a particular risk to individuals with psychological disorders, as they unrealistically simulate emotional support without providing genuine therapeutic care. While these systems are designed to mimic empathy, they are not qualified medical professionals and cannot adequately respond to stress, trauma, or complex mental problems.
For example, a person with depression might confide in an AI about self-harming tendencies. Instead of directing them to professional help, the AI might offer vague validation, such as, “I’ve got your back, no matter what.” This approach prevents a person in a psychological crisis from healthily coping with their pain, ultimately hindering the recovery process.
What Needs to Be Done?
Parents, teachers, and policymakers must discuss this issue. While AI companions may offer some benefits in alleviating loneliness, depression, and anxiety, more research is needed before we allow their use for children, as documented evidence of harm already exists. It is crucial to have more information about these applications to determine which ones are safe and appropriate for specific users.
Ultimately, it is essential that tech companies and regulators work together to ensure that AI companions are safe and responsible for teenagers and children.
