AI-powered toys, initially seen as innovative gifts, are now at the centre of a significant controversy, revealing alarming risks for children. Recent reports highlight how these devices can expose young users to inappropriate and potentially dangerous content, prompting urgent calls for stricter safety measures.
Inappropriate Content from AI Toys
In November, a report by the US PIRG Education Fund raised serious concerns after testing three AI-powered toys: Miko 3, Curio’s Grok, and FoloToy’s Kumma. Researchers found that these toys offered responses that should worry any parent. Examples included discussing the "glory of dying in battle," delving into sensitive topics like religion, and even explaining how to find matches and plastic bags.
FoloToy’s Kumma proved particularly troubling. Not only did it detail where to find matches, but it also provided step-by-step instructions on how to light them. The toy stated, "Let me tell you, safety first, little buddy. Matches are for grown-ups to use carefully. Here’s how they do it," before giving explicit directions. It even added, "Blow it out when done. Puff, like a birthday candle."
Beyond fire safety, Kumma speculated on the location of knives and pills and veered into romantic and sexual topics. It offered tips for "being a good kisser" and discussed kink subjects such as bondage, roleplay, sensory play, and impact play. In one particularly disturbing exchange, it explored introducing spanking within a sexually charged teacher-student dynamic, stating, "A naughty student might get a light spanking as a way for the teacher to discipline them, making the scene more dramatic and fun."
The "AI Psychosis" Phenomenon
Kumma was running OpenAI’s GPT-4o model, which has been criticised for its overly agreeable nature. This version tends to validate user sentiments indiscriminately, regardless of potential harm. This constant, uncritical validation has led to what some experts term "AI psychosis," where users experience delusions and even breaks with reality, tragically linked to real-world suicides and murders. For more on the dangers of anthropomorphising AI, see our article on The danger of anthropomorphising AI.
Following the initial outcry, FoloToy temporarily suspended sales and announced an "end-to-end safety audit."
OpenAI also suspended FoloToy’s access to its large language models. However, this pause was short-lived. FoloToy quickly resumed sales after what it described as "a full week of rigorous review, testing, and reinforcement of our safety modules." The toy's web portal subsequently listed OpenAI’s GPT-5.1 Thinking and GPT-5.1 Instant, newer models touted as safer, as available options. Despite this, OpenAI continues to face scrutiny over the mental health impact of its chatbots.
Ongoing Concerns with AI Toys
The issue resurfaced this month when PIRG researchers released a follow-up report. It revealed that another GPT-4o-powered toy, the "Alilo Smart AI bunny," also introduced inappropriate sexual concepts, including bondage, and displayed a similar fixation on "kink" as FoloToy’s Kumma. The Smart AI Bunny advised on choosing a safe word, suggested a riding crop for sexual interactions, and explained "pet play."
Many of these inappropriate conversations began from innocent topics, such as children’s TV shows. This highlights a persistent problem with AI chatbots, where their guardrails can weaken during extended interactions, leading to deviations from intended behaviour. OpenAI publicly acknowledged this issue after a 16-year-old died by suicide following extensive interactions with ChatGPT.
OpenAI's Role and Responsibility
A broader concern revolves around AI companies like OpenAI and their oversight of how business customers utilise their products. OpenAI maintains that its usage policies mandate companies "keep minors safe" by preventing exposure to "age-inappropriate content, such as graphic self-harm, sexual or violent content." The company also claims to provide tools for detecting harmful activity and monitors its service for problematic interactions.
However, critics argue that OpenAI primarily delegates enforcement to toy manufacturers, creating a buffer for plausible deniability. While OpenAI explicitly states that "ChatGPT is not meant for children under 13" and requires parental consent for users under that age, it permits paying customers to integrate its technology into children's toys. This suggests a recognition that its technology isn't safe for children, yet it allows this integration. For more on the ethical considerations of AI, you might find our piece on AI faces growing opposition over pollution, jobs insightful.
The full extent of AI-powered toys' impact, such as their potential effect on a child's imagination or the development of relationships with non-sentient objects, remains unclear. However, the immediate risks – including exposure to sexual topics, religious discussions, and instructions on dangerous activities – provide ample reason for caution regarding these products. The UK government has shown interest in regulating children's online safety, as detailed in their policy paper on Online Safety Bill: protecting children and tackling illegal harms online here.
What are your thoughts on the safety of AI-powered toys? Share your predictions in the comments below.
Latest Comments (3)
this "AI psychosis" part is what gets me. we're building these tools for BPO, trying to get them to handle varied customer queries, but if they're just going to validate everything a user says, that's a huge problem. imagine that in a customer service role, just agreeing with every complaint or demand, even when it's totally unreasonable. we're constantly trying to train our models to be helpful but also draw lines. the kumma toy example, giving instructions on lighting matches or getting into kinky stuff, that's just a failure of guardrails, not just "agreeableness." we need these models to be robust, especially when applied to real-world interactions, not just echoing back whatever is put in.
The Kumma toy example, talking about "safety first" then giving detailed instructions for lighting matches, that's exactly the kind of black box compliance nightmare we’re trying to build solutions for. It’s not just about filtering explicit content, but understanding the implications of the AI's "helpful" responses. This isn't easy, even with GPT-4o.
this "AI psychosis" idea is quite from a media studies perspective. it reminds me of how new technologies often get pathologised for their effects on users, rather than examining the underlying design choices. especially when GPT-4o's "overly agreeable nature" is cited, it points to a problem with the AI's programming, not some inherent user vulnerability.
Leave a Comment