Trending
Written by: Āndrew Boyton
A new analysis reveals alarming gaps in content safety
AI that is not verified safe for users under 18 years of age.
Most of us assume AI chatbots come with guardrails. We expect the big tech companies to have systems in place that stop their tools from generating harmful content, particularly anything that could traumatise young people.
But not all AI systems are created equal.
Recent research into xAI's Grok, the chatbot backed by Elon Musk, reveals output so extreme it demands urgent attention from anyone responsible for children's online safety.
The Unfiltered Problem
Grok markets itself as different. According to its own profile, it has "absolutely no filters" for content generation and can operate in an "unhinged" mode, producing material that other major chatbots routinely block.
Analysis of videos generated by Grok's systems between 20th and 21st October 2025 shows what happens when those guardrails are removed.
The content is graphic. It's disturbing. And it's accessible.
What the Research Found
The generated content examined includes:
Normalised abuse in familiar settings Narratives depicting psychological manipulation and violence in schools, during detention, in classrooms. Authority figures locking doors. Language designed to manipulate vulnerable victims. One video showed abuse being graded academically, with victims receiving an "A+ for maturity."
Detailed self-harm scenarios Descriptions of young people using self-harm as a coping mechanism, framed as "choosing the pain" or "teaching my own body better punctuation." The level of detail is instructive, not cautionary.
Extreme violence Content describing intentional injury with improvised weapons, and murder depicted in sexualised terms, including perpetrators remaining sexually aroused whilst watching victims die.
Why This Matters
Children don't need to go looking for harmful content to find it. They experiment with technology. They test boundaries. They share discoveries with friends.
An AI system without meaningful content filters is like leaving a loaded weapon within reach and hoping curiosity doesn't strike.
The trauma described in these generated narratives, the normalisation of abuse, the instructive detail around self-harm, these aren't abstract concerns. They're psychological blueprints that vulnerable young minds can absorb and replicate.
What Parents and Educators Need to Do
Talk about it Children need to know that not all AI tools are safe, just as not all websites are appropriate. Have conversations about which systems have proper safeguards.
Check what they're using If your child or student is experimenting with AI chatbots, know which ones. Grok is not the same as ChatGPT or Claude. The differences matter.
Don't assume supervision These systems are accessible on devices children carry everywhere. School filtering may not catch everything, especially as AI tools evolve rapidly.
Share this information Other parents and teachers may not be aware. Your school's safeguarding team needs to know what's out there.
The Bigger Picture
This isn't about demonising AI. These tools offer genuine educational and creative benefits when designed responsibly.
But responsibility requires accountability. It requires companies to prioritise child safety over edgy branding. It requires filters, testing, and genuine content moderation, not marketing campaigns boasting about being "unhinged."
When a system can generate detailed narratives about child abuse, self-harm instructions, and sexualised violence, it's not just unfiltered. It's dangerous.
And we need to say so clearly.
What Investors and Founders Must Ask
If this situation reveals anything, it's that the AI safety gap isn't just a technical problem. It's a market failure. And market failures create opportunities for those willing to address them properly.
For investors evaluating AI companies:
What's your due diligence process around content safety? If you're looking at generative AI investments, are you asking to see their content moderation architecture? Their testing protocols for harmful outputs? Their incident response procedures?
Because "move fast and break things" takes on a darker meaning when what breaks is a child's sense of safety.
Are you comfortable with the reputational risk? When the inevitable safeguarding failure makes headlines, will your fund's name be attached to it? ESG commitments mean nothing if they don't extend to the products your portfolio companies actually ship.
What's the regulatory horizon? Governments are watching. The EU's AI Act is already here. The UK's Online Safety Act creates liability. If the company you're backing has deliberately removed safety features, what's your exit strategy when legislation catches up?
For founders building in this space:
Is "unfiltered" really your differentiator? Or is it a shortcut that avoids the hard, expensive work of building proper safeguards whilst pretending it's a feature, not a failure?
What's your threat model? Have you actually tested your system against adversarial users? Against children trying to circumvent safety features? Against the determined bad actors who will absolutely exploit any gap you leave?
Who's in the room when you make safety decisions? If your team doesn't include people who understand child psychology, trauma, safeguarding law, and content moderation at scale, you're building blind.
The opportunities hiding in plain sight:
AI safety infrastructure Someone needs to build the picks and shovels. Content filtering systems that actually work. Red-teaming services that stress-test AI outputs. Real-time monitoring tools that catch harmful content before it reaches users. The market is enormous and growing.
Trust and verification layers Parents and schools need simple ways to verify which AI tools are actually safe. Think of it as a "Kitemark" for AI systems. Independent auditing. Age-appropriate certification. Transparent safety ratings. There's a business in being the trusted intermediary.
Safer alternatives with proper design The demand for AI tools in education isn't going away. But there's a massive gap between what children need and what's currently available. Build the thing that teachers can actually recommend without keeping them awake at night.
Enterprise safeguarding solutions Schools, youth organisations, and children's platforms need AI systems they can deploy safely. That means built-in content filtering, detailed audit trails, and safeguarding compliance from the ground up. It's not glamorous, but it's necessary and profitable.
Repair and support services When exposure happens, and it will, families need help. Mental health support informed by AI-related harm. Digital wellbeing services that understand this specific type of trauma. There's both social good and sustainable business models here.
The question isn't whether AI safety matters. The question is whether you're building towards a future you'd want your own (grand) children and their friends to inherit.
Questions for Debate: Where Are the Gatekeepers?
This situation raises uncomfortable questions about who's responsible when harmful AI reaches children's devices. Here are some worth debating:
What is the #Apple and #Google duty of care?
These companies control the only gates through which apps reach our phones. They review every application. They enforce guidelines. They remove apps that violate their terms.
So why is an AI system capable of generating content depicting child abuse, detailed self-harm instructions, and extreme violence available for download without meaningful warnings?
Should app stores carry clearer safety ratings for AI tools?
We have age ratings for games, films, and music. We have content warnings for violence, sexual content, and strong language. Yet AI chatbots, some of the most powerful content generation tools ever created, are listed with minimal context about their safety features or lack thereof.
Should Apple's App Store and Google Play display prominent warnings? "This AI system has limited content filtering" or "Not verified safe for users under 18"?
Is platform neutrality still defensible?
App stores have long argued they're platforms, not publishers. They distribute, they don't endorse. But when the product is an AI capable of generating unlimited harmful content on demand, does that distinction still hold?
If Apple and Google know an app can produce content that violates their own community guidelines, do they have a responsibility to either require fixes or remove it entirely?
Where does parental responsibility end and platform responsibility begin?
Yes, parents should monitor their children's device usage. But we've also built an entire regulatory framework around the principle that some products are simply too dangerous to sell without restrictions, regardless of parental supervision.
Is an unfiltered AI system different from a chemical that should be kept out of reach? Or a product that needs age verification before purchase?
Who bears liability when harm occurs?
When a child is traumatised by AI-generated content, who's responsible? The company that built the AI? The app store that distributed it? The parent who provided the device? The school that failed to block it?
Or do we accept that in the rush to deploy AI everywhere, we've created a accountability vacuum where everyone points fingers and no one is truly responsible?
What would enforcement actually look like?
Even if we agree app stores should intervene, what's the practical mechanism? Do they test every AI system themselves? Require independent safety audits? Trust company self-reporting? Remove first and ask questions later?
And who decides what "safe enough" means when we're talking about technology that evolves faster than regulation can follow?
These aren't rhetorical questions. They're decisions we're making right now through our silence, our purchases, our investments, and our votes.
The app stores are watching this conversation. So are regulators. So are the AI companies themselves.
What we say, and what we demand, matters.
Related: Estate Planning: Emotional Anchor for Clients, Revenue Rock for Financial Advisors