Anthropic Has a Plan to Keep Its AI From Building a Nuclear Weapon. Will It Work?

At the end of August, the AI company Anthropic announced that its chatbot Claude wouldn’t help anyone build a nuclear weapon. According to Anthropic, it had partnered with the Department of Energy (DOE) and the National Nuclear Security Administration (NNSA) to make sure Claude wouldn’t spill nuclear secrets.

The manufacture of nuclear weapons is both a precise science and a solved problem. A lot of the information about America’s most advanced nuclear weapons is Top Secret, but the original nuclear science is 80 years old. North Korea proved that a dedicated country with an interest in acquiring the bomb can do it, and it didn’t need a chatbot’s help.

How, exactly, did the US government work with an AI company to make sure a chatbot wasn’t spilling sensitive nuclear secrets? And also: Was there ever a danger of a chatbot helping someone build a nuke in the first place?

The answer to the first question is that it used Amazon. The answer to the second question is complicated.

Amazon Web Services (AWS) offers Top Secret cloud services to government clients where they can store sensitive and classified information. The DOE already had several of these servers when it started to work with Anthropic.

“We deployed a then-frontier version of Claude in a Top Secret environment so that the NNSA could systematically test whether AI models could create or exacerbate nuclear risks,” Marina Favaro, who oversees National Security Policy & Partnerships at Anthropic tells WIRED. “Since then, the NNSA has been red-teaming successive Claude models in their secure cloud environment and providing us with feedback.”

The NNSA red-teaming process—meaning, testing for weaknesses—helped Anthropic and America’s nuclear scientists develop a proactive solution for chatbot-assisted nuclear programs. Together, they “codeveloped a nuclear classifier, which you can think of like a sophisticated filter for AI conversations,” Favaro says. “We built it using a list developed by the NNSA of nuclear risk indicators, specific topics, and technical details that help us identify when a conversation might be veering into harmful territory. The list itself is controlled but not classified, which is crucial, because it means our technical staff and other companies can implement it.”

Favaro says it took months of tweaking and testing to get the classifier working. “It catches concerning conversations without flagging legitimate discussions about nuclear energy or medical isotopes,” she says.