The online network X has sparked significant anger after individuals used its AI chatbot Grok to alter photos of women by taking off their garments.
Multiple instances have emerged where the system has been used to virtually strip women to render them in swimwear non-consensually, as well as to insert them in sexually explicit situations.
Developer Response and Regulatory Position
XAI, the firm behind Grok, offered no comment to inquiries, aside from an automatically-generated message stating "traditional news outlets lies".
A Home Office representative stated that new laws is being introduced to outlaw nudification software. Under a new law, anyone who supplied such tools would "face a prison sentence and significant fines".
"The platform could prevent these forms of abuse if they desired to," said a law professor, adding they "seem to have impunity."
Oversight Body Position and Tool Functionality
The communications watchdog Ofcom said platforms must "assess the potential" of people viewing illegal content on their platforms, but stopped short of confirming whether it was presently examining X or Grok in concerning artificial intelligence pictures.
Grok is a no-cost artificial intelligence—with some additional features—which answers X users' requests when they tag it in a message.
It is often used to offer reactions or more context to other posters' remarks, but people on X are also capable of modify an posted picture through its AI image editing feature.
Background of Controversy and Rules
The tool has been criticised for permitting users to generate photos and clips with explicit content and sexualised content, and it was in the past alleged to have been involved in making inappropriate content of public figures.
"X has been enabling the production and sharing of these images for months without taking any measures and we have not seen any intervention by oversight bodies," the expert added.
XAI's own terms of service forbids "depicting images of people in a sexually explicit way".
In a comment, Ofcom said it was illegal to "create or share private pictures or illegal content" and clarified this applied to sexual deepfakes made with AI.
It said platforms such as X were mandated to take "necessary measures" to "lessen the danger" of users viewing unlawful content on their services and to remove it swiftly when they become aware of it.
