A woman made her AI voice clone say “arse.” Then she got banned.

“Well, apparently not, because the next day a human banned me!!!!” says Joyce. She says she felt mortified. “I’d just got my voice back and now they’d taken it away from me … and only two days after I’d done a presentation to my local MND group telling them how amazing ElevenLabs were.”

Joyce contacted ElevenLabs, who apologized and reinstated her account. But it’s still not clear why she was banned in the first place. When I first asked Sophia Noel, a company representative, about the incident, she directed me to the company’s prohibited use policy.

There are rules against threatening child safety, engaging in illegal behavior, providing medical advice, impersonating others, interfering with elections, and more. But there’s nothing specifically about inappropriate language. I asked Noel about this, and she said that Joyce’s remark was most likely interpreted as a threat.

ElevenLabs’ terms of use state that the company does not have any obligation to screen, edit, or monitor content but add that it may “terminate or suspend” access to its services when content is “reasonably likely, in our sole determination, to violate applicable law or Terms.” ElevenLabs has a moderation tool that “screens content to ensure it aligns with our Terms of Service,” says Dustin Blank, head of partnerships at the company.

The question is: Should companies be screening the language of people with motor neuron disease?

After all, that’s not how other communication devices for people with this condition work. People with MND are usually advised to “bank” their voices as soon as they can—to record set phrases that can be used to create a synthetic voice that sounds a bit like them, albeit a somewhat robotic-sounding version. (Jules recently joked that his sounded like “a Daft Punk song at quarter speed.”)

Banked voices aren’t subject to the same scrutiny, says Joyce’s husband, Paul. “Joyce was told … you can put whatever you want in there,” he says. Voice banking wasn’t an option for Joyce, whose speech had already deteriorated by the time she was diagnosed with MND. Jules did bank his voice but doesn’t tend to use it, because the voice clone sounds so much better.

Joyce doesn’t hold a grudge—and her experience is far from universal. Jules uses the same technology, but he hasn’t received any warnings about his language—even though a comedy routine he performs using his voice clone contains plenty of curse words, says his wife, Maria. He opened a recent set by yelling “Fuck you guys!” at the audience—his way of ensuring they don’t give him any pity laughs, he joked. That comedy set is even promoted on the ElevenLabs website.