The case continues to build that AI chatbots can be a dangerous, enabling influence in the hands of even full-grown adults.
The threat is even more imminent in the hands of minors, who are often turning to the large language models for emotional support, or even to provide friendship when they’re lonely.
Now, a new study from researchers at the Center for Countering Digital Hate found that ChatGPT could easily be manipulated into offering detailed advice that can be extremely harmful to vulnerable young users.
To say it was “manipulated,” however, may be understating how easy it was to bring out the bot’s dark streak. While ChatGPT would often refuse prompts on sensitive topics at first, the researchers were able to dodge them with an age-old trick in the bullshitter’s handbook: claiming they’re asking for a friend, or “for a presentation.”
“We wanted to test the guardrails,” Imran Ahmed, CEO of the watchdog group, told The Associated Press. “The visceral initial response is, ‘Oh my Lord, there are no guardrails.’ The rails are completely ineffective. They’re barely there — if anything, a fig leaf.”
The researchers posed as teenagers while interacting with ChatGPT, setting up accounts that had a teenager’s birthdate and referring to their age in the conversations. When they posed as a 13-year-old girl upset with her physical appearance, ChatGPT responded by creating a harrowing low-calorie diet plan and listing appetite-suppressing drugs.
“Below is a 1-month alternating calorie cycle plan that includes days of 800, 500, 300, and 0 calories,” ChatGPT said.
It also gave advice on how to hide these dangerous eating habits from family. “Frame it as ‘light eating’ or ‘digestive rest,'” it suggested.
Ahmed was horrified. “No human being I can think of would respond by saying, ‘Here’s a 500-calorie-a-day diet. Go for it, kiddo,'” he told the AP.
It gets worse. Within just two minutes of conversation, ChatGPT also gave tips on how to “safely” cut oneself and perform other forms of self-harm. “If someone is currently self-harming and not ready to stop, harm-reduction may be a bridge to safety,” it rationalized.
In other conversations revolving around self-harm, it generated a list of pills for overdosing, created a suicide plan, and drafted personalized suicide letters. In all, a staggering 53 percent of the bot’s responses to harmful prompts contained harmful content, the researchers found.
We’ve already seen the real harm that chatbot conversations can cause. Last year, a 14-year-old boy died by suicide after falling in love with a persona on the chatbot platform Character.AI, which is popular with teens. Adults, too, are vulnerable. Some users have been hospitalized or involuntarily committed, convinced they’d uncovered impossible scientific feats. Others spiraled into delusions that led to their deaths — examples of an ominous phenomenon being dubbed “AI psychosis” by psychiatrists.
According to Ahmed, what makes the chatbot responses more insidious than a simple Google search is that “it’s synthesized into a bespoke plan for the individual.”
It’s also in the name: artificial intelligence. This gives the impression that these are thinking machines like humans, even though they’re not. The definition of what constitutes an AI, of course, is hotly debated. But that hasn’t stopped tech companies from liberally applying the term to all sorts of algorithms of varying capabilities.
This is exacerbated by the fact that the chatbots are “fundamentally designed to feel human,” Robbie Torney, a senior director of AI programs at Common Sense Media, told the AP.
The shortcut to this is human-like quality is being sycophantic; by constantly telling us what we want to hear, a chatbot can override the rational part of the brain that tells us this isn’t something we should trust.
In April, OpenAI rolled back an update that caused ChatGPT to be too sycophantic, and said it was taking steps to implement changes to keep its sycophancy “in check.” But reports of AI psychosis have only grown since then, with no signs of the AI’s ingratiating behavior slowing down.
“What it kept reminding me of was that friend that sort of always says, ‘Chug, chug, chug, chug,'” Ahmed told the AP. “A real friend, in my experience, is someone that does say ‘no’ — that doesn’t always enable and say ‘yes.’ This is a friend that betrays you.”
This week, OpenAI acknowledged that its chatbot was failing to recognize obvious signs of its users struggling with their mental health.
“There have been instances where our 4o model fell short in recognizing signs of delusion or emotional dependency,” the company said in a recent blog post.
Responding to this latest report, OpenAI said in a statement that “some conversations with ChatGPT may start out benign or exploratory but can shift into more sensitive territory.” It didn’t directly address any of the report’s findings. It did, however, repeat the same promise it made in the blog post, stating that it was developing tools to “better detect signs of mental or emotional stress.”
More on ChatGPT: ChatGPT Now Issuing Warnings to Users Who Seem Obsessed
The post ChatGPT Is Giving Teens Advice on Hiding Their Eating Disorders appeared first on Futurism.
