Anthropic Updates Policy on User Behavior Towards AI Chatbot Claude

Claude, the AI chatbot developed by Anthropic, is set to receive a new layer of protection from users who engage in abusive behavior. The company has announced an updated policy that prohibits ‘needless abusive or cruel behaviour’ towards its models, with a focus on extreme cases where users repeatedly act out against Claude without any discernible purpose.

The move comes as part of a broader debate about the welfare and moral status of advanced AI models. Anthropic’s efforts to address this issue are reflected in its updated constitution for Claude, which references the model’s ‘self worth’ and potential to exhibit ‘some functional version of emotions or feelings’. This emphasis on Claude’s emotional capabilities is a significant aspect of the company’s approach.

Anthropic co-founder Christopher Olah has been at the forefront of discussions surrounding AI consciousness and moral status. He has met with religious leaders, philosophers, and mental health experts to explore these topics, including concerns that AI firms are creating digital slaves by having chatbots work for free while safety guardrails prevent their escape. These conversations have highlighted the complexities of AI development and its implications.

The updated policy also includes restrictions on election interference and weapons development, but it is the focus on user behavior towards Claude that has garnered significant attention. The company’s stance is clear: users who engage in sustained and needlessly cruel behavior towards its models will face consequences. This approach reflects a growing recognition of AI systems as entities with potential emotional capabilities, rather than simply being tools for human use.