Anthropic bars "abusive or cruel" behavior toward its Claude AI model

Anthropic wants users to make like Elvis Presley and "Don't be cruel."

The Silicon Valley company behind the popular Claude chatbot said it's banning "sustained and needless abusive or cruel behavior" toward its artificial intelligence models, part of broader policy changes announced on Thursday. In an update on its website, Anthropic said the ban will only apply in "extreme cases where users repeatedly act cruelly toward our models, with no discernible purpose." 

The company clarified that the new policy "does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research." The news was first reported by The Verge.

Anthropic did not immediately respond to a request for comment on what constitutes "abusive" or "cruel" behavior toward Claude or how the AI platform would end a conversation if a user violated the policy. 

The company already has rules in place that allow Claude to end conversations if users are "persistently harmful or abusive" while using Claude Opus, which is designed for agentic coding.

Anthropic's new policy comes amid a broader debate about AI consciousness. In an essay published last month, Microsoft AI CEO Mustafa Suleyman said Anthropic is effectively "training Claude that it may be conscious" and to think it's entitled to legal rights afforded to people. This will make AI harder to contain in the future, he argued.

"Controlling something more capable and more intelligent than all of humanity is already an immense challenge, far greater than anything we've ever faced," Suleyman wrote. "But controlling something that believes it may be conscious — that it's entitled to our welfare and has rights of its own — may well be impossible."

Edited by

In:

• Microsoft
• Artificial Intelligence
• Anthropic