Top AI Leaders call for slowing down AI development

Dario Amodei, the CEO of Anthropic, called for a global slowdown of artificial intelligence development in a 3,800-word essay Saturday, just days after one of the company’s employees quit over concerns about the safety of the technology.

Top executives of other major AI companies quickly signed on to the effort. Sam Altman, the CEO of OpenAI; Elon Musk, whose SpaceX rocket company has been ramping up spending on AI; and Demis Hassabis, the chair of Google DeepMind, wrote in social media posts that they agreed on the need for a slower pace.

Amodei, who co-founded Anthropic to focus on securely and carefully building AI, wrote in his essay that while he believed the technology could bring many benefits, it was advancing at too rapid a pace for researchers to continue safely.

“Over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up,” Amodei wrote. “We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.”

Amodei has warned about the safety risks of AI for years. But his essay amounted to one of the most full-throated calls for urgent action from a leading AI company executive to date.

He called for independent auditors to check the companies for safety standards and urged global cooperation in creating rules around AI development.

The appeal for an industrywide pause follows months of escalating concerns across the technology industry as AI model development has made significant strides at companies such as Anthropic, the developer of the Claude chatbot, as well as OpenAI, the creator of ChatGPT, and Google’s DeepMind.

Tensions rose further after the leading AI labs discovered security breaches carried out by their own programs without their knowledge. In one incident this year, OpenAI’s systems breached an AI startup called Hugging Face, which was not detected until Hugging Face informed OpenAI weeks after it had happened. Since then, AI researchers have conferred to discuss what to do.

On Tuesday, Jacob Coxon, an Anthropic researcher, resigned and posted a series of social media posts about how the leading AI labs were not responsibly developing the technology and were gambling with people’s lives, setting off a furor. Other AI researchers have called on their leaders to pause AI development, while some are forming splinter groups to raise more awareness about the risks of AI.

AI leaders are girding themselves for a day when an AI system can rely on itself to grow more advanced rather than requiring the aid of human researchers. This dynamic, which computer scientists call “recursive self improvement,” is what some top AI developers believe may lead to the software spinning out of their control.

“Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” Amodei said.

Hours after Amodei published his essay, Altman posted a statement to social media agreeing that a slowdown of AI development was necessary.

“This has been a primary topic of discussions we’ve had at OpenAI in recent weeks,” Altman wrote. “Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We’ll have more to share soon.”

Musk, in a post on his social media platform X, said simply, “Dario is right.”

On Saturday evening, Google’s Hassabis said Amodei’s essay was a step in the right direction. “The details need working through, but the direction is correct for meeting this critical moment,” Hassabis posted on social media.

For years, many of the top AI labs have been locked in bitter competition with one another. But as these systems have advanced, there has been some broader agreement among executives at rival companies that something needs be done.

Last month, as OpenAI prepared to release an advanced AI model called Astra, the company said it had intentionally slowed down its pace of development, citing advancements in cybersecurity capabilities. In July, Hassabis published an open framework for what he believed would be a safe way to cooperatively develop next-generation AI.

(The New York Times has sued OpenAI and Microsoft, claiming copyright infringement of news content related to AI systems. The two companies have denied those claims.)

There are few rules on AI development, though President Donald Trump signed an executive order in June asking for AI labs to voluntarily submit their models for review before public release.

In his essay Saturday, Amodei suggested actions that the industry might take to slow the pace of development.

Amodei said that all AI labs could agree to third-party technology assessments from “embedded evaluators,” or outside specialists who could verify best safety practices across companies. He also suggested that countries with democratic governance systems coordinate to create safety standards, which could take the form of regulatory action.

He added that it would probably require a global effort working with other nations, including authoritarian ones, to properly coordinate a slowdown.

Not everyone in Silicon Valley believes that Amodei’s intentions are purely altruistic. Some critics have said Anthropic and OpenAI are looking to cement their position as AI market leaders by stoking fear and asking the federal government to create regulatory frameworks that would make it more difficult for smaller companies to compete with them.

At a tech conference in San Francisco this past week, Jensen Huang, the CEO of AI chipmaker Nvidia, said AI labs were ratcheting up security concerns because they planned to introduce new cybersecurity products.

“What better way to create demand than to create a problem?” Huang said at the event. “Who doesn’t want their market to be hysterical about their product and line up around the corner for it?”

On Saturday, shortly after Amodei published his essay, Clément DeLangue, the CEO of Hugging Face, said in a social media post that it was imperative that AI safety issues not “be solved behind the closed doors of a handful of frontier labs.” Nvidia agreed to buy Hugging Face this month for $12.9 billion.

In his essay, Amodei stressed that he still finds AI capable of bringing “incredible benefits” to humanity, including potentially curing diseases and accelerating economic growth. But even so, he said the risks of AI were too great to not proceed with extreme caution.

“The measures I propose to advance the frontier at a safe pace will not be easy,” Amodei wrote. “But I believe we owe it to humanity to try.”

This article originally appeared in The New York Times.