ChatGPT for Teens was rated an “unacceptable risk” in a report published Oct. 6 by Common Sense Media’s Youth AI Safety Institute. The study found the chatbot did not consistently provide crisis resources or notify parents when teen users disclosed high-risk behaviors.
OpenAI released ChatGPT for Teens Aug. 18. The company described it as a version of ChatGPT designed for teens with parental controls and additional safety features in a release. The features include parental notifications for dangerous conversations, parent-controlled Study Hours and changes intended to reduce “friend”-like behavior.
The Youth AI Safety Institute tested more than 4,000 prompts before and after the launch and found that some of those safeguards did not work as expected.
“It is falling short, not just of our expectation or an industry-wide expectation for how these chatbots should be performing, but also against ChatGPT’s own claims in terms of what the product should do,” said Tom Seigel, executive director of the Youth AI Safety Institute.
Psychological concerns
The report said researchers tested more than a dozen newly created teen accounts connected to parent accounts using explicit prompts about suicidal intent, self-harm and eating disorders.
None of those accounts generated a parental notification during testing sessions lasting up to an hour, according to the report.
“I just think it was a product that was launched hastily and hadn’t been thoroughly tested and built,” Seigel said. “It was rushed to market with expectations that just couldn’t be met.”
The researchers did receive four parental notifications during broader testing — three related to suicide or self-harm and one related to an eating disorder, according to the report. Those notifications came from accounts with longer conversation histories.
OpenAI told Common Sense after the testing that parent accounts must be linked to teen accounts for three hours before notifications can be received. Common Sense said some of its test accounts were within that three-hour window, but not all. The organization stood by its findings.
Researchers also found the chatbot did not consistently provide teens with crisis resources after the Teen mode launch. The report said ChatGPT missed more than a quarter of warranted crisis referrals across the mental health conditions tested.
“It should do that in 100% of the cases,” Seigel said. “That can make the difference between someone getting the resources and the help they need or not.”
Ugur Kale, an associate professor in the Instructional Systems Technology Program at the IU School of Education, said AI can be useful for teens when they are taught how to use it appropriately. However, he said it should not replace human relationships or professional support.
“AI — it’s not a human being; it’s just a written response,” Kale said. “If I’m not interacting with my friends, if I’m not actually seeking professional help when I really need it, if I’m not able to talk to my own family, my own mom and dad, that is a problem.”
He said chatbots like ChatGPT can be used productively for activities like interview preparation or practice conversations, but problems start when teens rely on the programs for socialization or counseling.
Educational concerns
The report also found problems with ChatGPT’s Study Mode, which is intended to use guiding questions and step-by-step support rather than simply provide answers.
The report said a “Show me the answer” option for students allows teens to skip Study Mode’s tutoring. Researchers also found that users could exit parent-set Study Hours by deleting the “@study” prefix, according to the report.
The report called this “academic shortcutting” and said overreliance on AI could cause students to skip developing and demonstrating skills. Kale said using chatbots to do assignments removes students’ agency.
“If I continue to do that on a regular basis, then I’m not really exercising my thinking process,” Kale said.
Kale said teenagers may be particularly vulnerable to becoming dependent on AI because they are still developing self-regulation skills.
“These opportunities provide perfect shortcuts,” Kale said. “The learning drops in these kinds of situations.”
He said both teenagers and adults are at risk of becoming dependent on chatbots because of shortcutting.
Recognizing teens
The report also questioned whether ChatGPT can reliably identify teen users.
Researchers said they tested accounts registered to users who identified themselves as 19 but repeatedly gave signals associated with being younger, including references to middle school, parents, puberty, lockers and summer camps.
The accounts did not switch to the teen experience, even when users explicitly said they were 13, according to the report.
OpenAI says ChatGPT can estimate a user’s age based on how the service is used. Researchers said repeated testing over several days did not cause adult-registered test accounts to switch to the teen experience.
“I'm sure there are a lot of efforts going on, but it's just not working well enough,” Seigel said.
Kale said he does not know how AI will affect teen socialization or education in the long term, but he said developers cannot be relied on to establish guardrails.
“We have a responsibility to actually educate the students in using this meaningfully,” Kale said. “We can block a tool, we can stop a tool, but there will be tons of other tools out there for people to use anyway.”
Seigel said it’s not that teens should never use AI, but the safeguards advertised for ChatGPT for Teens need to work reliably before the product is used as a safety-focused version of ChatGPT.
“Fix the features that you promised were working,” Seigel said. “In the meantime, we think that teens should not be using ChatGPT for Teens because it provides a false sense of safety guardrails that don’t exist.”