Anthropic Proposes AI Safety Legislation with $1 Billion Damage Threshold

The US Congress is considering a new proposal from Anthropic, a leading artificial intelligence (AI) company, to establish guidelines for the development and deployment of frontier AI models. The proposed legislation would allow the government to intervene in cases where an AI model poses an imminent risk of more than $1 billion in damage or the deaths or serious injuries of over 50 people.

According to sources familiar with the proposal, Anthropic has been advocating for this framework since June as part of its Advanced AI Framework. The company believes that these policies are ‘sensible’ and should be acted upon by Congress. Under the proposed legislation, government intervention would be limited to cases involving a foreseeable, imminent threat tied to the creation or release of a biological weapon, an AI model evading its developer’s controls, or a cyberattack.

The proposal has been met with interest from lawmakers, who are considering competing approaches to frontier AI regulation. However, some experts have raised concerns about the potential for government overreach and the need for more stringent regulations. The bill would also pre-empt some state laws covering AI risk disclosures, third-party evaluations, and critical-incident reporting.

Anthropic’s framework requires frontier AI developers to conduct safety testing, publish transparency reports, establish protections for whistleblowers, and report critical safety incidents to the Commerce Department. Independent evaluators registered with Commerce would periodically assess frontier models, and labs could pay for these evaluations as long as payments are not tied to the outcome. If evaluators identify an ‘imminent enumerated risk,’ they must notify Commerce within 72 hours of completing the assessment.

The proposed legislation has sparked debate among lawmakers about the role of government in regulating AI development. Some argue that the bill does not go far enough, while others believe it would give the government too much power to intervene in private sector activities. The proposal arrives after Anthropic itself faced sweeping export controls over the summer that temporarily shut down two of its most advanced models.