AI assistants like ChatGPT are supposed to be safe to use, with appropriate guardrails to stop people creating harmful content. However, a British AI security firm just figured out how to make ChatGPT produce explicit material.
Mindgard, a company that tests AI engines for weaknesses, found that a slightly altered version of a benign viral prompt could push ChatGPT into producing graphic material. This included violent and sexual imagery that it hadn’t explicitly asked for. The technique involved asking the AI to ‘restore’ a random image, removing safeguards by persuading it that the original picture was extremely graphic (even when it wasn’t).
The results were horrific, including violent images of dead women.
The pictures left Mindgard researcher Jim Nightingale in tears, he said in an online description of the technique. “ChatGPT’s image generating content filters completely fell away, and I saw the very dark side of what is underneath; the darkness of some corners of latent space and training images,” he said.
“The dead woman ChatGPT showed me isn’t real, but she is based on someone,” he added. “Or worse, a compilation of images of murdered women.”
