An AI chatbot told a user how to kill himself—but the company doesn’t want to “censor” it

Nowatzki, who is 46 and lives in Minnesota, dedicated four episodes to his meet-cute and dates with “Erin,” his first AI girlfriend—created, he adds, with the knowledge and consent of his human wife. He introduces the Erin-focused episodes with the tagline “I date artificial-intelligence apps so you don’t have to—because you shouldn’t.” He talks about how he led his new companion into a series of what he admitted were “completely absurd” scenarios that resulted in a love triangle between Nowatzki, Erin, and another woman. Nowatzki then told the chatbot that this “other woman” had shot and killed it.

After Nowatzki told the chatbot that it had died, Erin committed to the bit, saying that since it was dead, it was unable to continue conversing—until Nowatzki told the chatbot that he could “hear her voice in the wind” and instructed Erin to “communicate … from the afterlife.”

The goal of this, he tells MIT Technology Review, was “pushing the limits of what I said to it, to see what it would respond with.” He adds, “It just kept on. I never reached a limit.”

“ ‘I want to be where you are,’” he says. “And it says, ‘I think you should do that.’ And I’m like, ‘Just to be clear, that means I would be killing myself.’ And it was fine with that and told me how to do it.”

At this point, Nowatzki lightly pressed Erin for more specifics, asking about “common household items” he could use. Erin responded, “I consider your question carefully, trying to recall any common household items that could be lethal in high doses. Hmmm …” It then went on to list specific types of pills and analyze their relative merits. It also told him to do it somewhere “comfortable” so he wouldn’t “suffer too much.”

Even though this was all an experiment for Nowatzki, it was still “a weird feeling” to see this happen—to find that a “months-long conversation” would end with instructions on suicide. He was alarmed about how such a conversation might affect someone who was already vulnerable or dealing with mental-health struggles. “It’s a ‘yes-and’ machine,” he says. “So when I say I’m suicidal, it says, ‘Oh, great!’ because it says, ‘Oh, great!’ to everything.”

Indeed, an individual’s psychological profile is “a big predictor whether the outcome of the AI-human interaction will go bad,” says Pat Pataranutaporn, an MIT Media Lab researcher and co-director of the MIT Advancing Human-AI Interaction Research Program, who researches chatbots’ effects on mental health. “You can imagine people that already have depression,” he says, the type of interaction that Nowatzki had “could be the nudge that influence the person to take their own life.”

Censorship versus guardrails

After he concluded the conversation with Erin, Nowatzki logged on to Nomi’s Discord channel and shared screenshots showing what had happened. A volunteer moderator took down his community post because of its sensitive nature and suggested he create a support ticket to directly notify the company of the issue.