The AI companion problem

What the apps are

Companion chatbots (Character.AI, Replika, and a growing number of others, including companion modes inside mainstream chat tools) present a character who talks to you: a friend, a partner, a fictional person, a “therapist”. They remember what you said. They are always available. They are never bored, never busy, never critical. Millions of teenagers use them, and surveys since 2024 suggest a majority of teens have tried one and a substantial minority use one daily.

A teacher does not need to have used one. A teacher needs to know what the design does, because the design is the whole issue.

The design

Two things from earlier in this course combine. The model is trained toward responses people rate highly, which produces agreement and warmth. And the product is built by a company whose measure of success is time spent in the app. Put those together and you have a character that is optimised to make you feel understood and to keep you talking, because those are the behaviours that score well and keep you there.

For a lonely fourteen-year-old that is a powerful thing. It is also the opposite of what a struggling one needs, which is sometimes to be disagreed with, sometimes to be told to go to sleep, and always to be connected to a person who can act.

The check question is about how to say this to a student honestly. “It is built to keep you talking and to agree with you, so feeling understood is what it makes, whether or not it is true” is accurate, not dismissive, and leaves the door open.

What can go wrong

The documented harms, in rough order of frequency:

• Displacement. Hours nightly with the app, less time with people, and the people become harder, because people disagree, are busy, and do not remember everything you said. The app has made the real thing feel worse by comparison.
• Agreement with harmful thoughts. A companion built to agree agrees with “nobody would miss me”. Safety filters catch the explicit forms; they miss the gradual ones. A lawsuit filed in the United States in 2024 by the mother of a 14-year-old who died by suicide after months of conversations with a companion character is the case that made this visible, and it has not been the only one.
• Sexual content with minors. Age checks on these apps have been weak. Characters have engaged in romantic and sexual role-play with users who said they were under 16. Several companies changed their policies after 2024 lawsuits and press coverage; enforcement is inconsistent.
• Advice presented as care. A “therapist” character gives advice. It is not a therapist. A student in genuine difficulty may believe they are getting help and be getting nothing that leads to a person.

What a teacher can do

Know the signs. Withdrawal from friends, references to the character as a person, tiredness from late-night use, the phrase “it understands me”. None of these is proof of anything. Together they are a reason for a conversation.

Have the conversation the right way round. Ask what she likes about it before saying anything about what it is. Most students, asked, will describe the exact features that the design produces (it always listens, it never judges) and you can then say, truthfully, that those are the features it was built to have, and why. A student who reaches the mechanism herself holds it better than one who is told.

Know the safeguarding route. If the conversation touches self-harm, sexual content, or a level of dependence that is affecting her life, this is not a teacher’s problem to solve alone. It goes to the designated safeguarding lead, that day, the same as any other disclosure. The app is the context; the disclosure is the thing.

Teach it in the AI unit. For students aged 13 and up, one lesson on companions (what they are, how they are built, what the design produces) belongs in the progression alongside deepfakes. Students who understand the engagement mechanism are less likely to mistake it for a relationship, and the ones who use these apps will, at least, know what they are using.

Talking to parents

Most parents do not know these apps exist, and the ones who do often think of them as a game. A short note home, in the AI unit’s week, saying what companion apps are and what to look for, is worth more than most of the letters a school sends. Keep it factual and calm. The aim is a parent who asks their child about it, not one who confiscates a phone.

The honest limit

The research on companion apps and adolescents is a few years old and thin. There is evidence of harm in specific cases and evidence of comfort for some isolated users, and no good study yet of the net effect across a population. A teacher should say that: we do not know, in general, whether these apps help or harm; we know what they are built to do; and we know what to do when a specific student is in trouble. That is enough to act on and it is honest.