Are top artificial intelligence companies racing to create a dangerous technology that could kill many millions of people — or even wipe out the human race entirely?

Fears like these have long circulated in Silicon Valley and among niche communities obsessed with AI. In fact, several of the top AI companies were founded by leaders who each claimed they would take worries about “existential risk” more seriously than their rivals.

But as OpenAI, Anthropic, and other companies’ AI projects have advanced in recent years, the general public’s anxieties have tended to be less apocalyptic. Concerns about how AI will impact jobs, the environment, energy costs, or our politics have dominated the discourse, with the prospect of human extinction seeming rather far-fetched.

This was the week that worries AI might kill us all finally went mainstream — captivating public attention and even starting to galvanize politicians into more aggressive stances. An unprecedented rogue AI incident, a high-profile resignation from a top AI company, and increased concern from tech researchers about AI improving itself too quickly to be contained, have all combined to draw a much louder response from elected leaders in both parties.

The thing is, though, that politicians have already talked about this issue. The Biden administration put out an enormous executive order to limit the harms of AI, which President Donald Trump rescinded. The last two Congresses have promised different AI working groups, which have largely fizzled, or gone quiet. And even the Trump White House has a version of an AI safety plan, which it is enforcing — though the details are all kept private.

In conversations with AI safety advocates and experts this week, nobody I spoke to seemed confident that politicians were close to addressing the problem at the scale that AI really represents — and a lot of them aren’t sure the government could address it, even if it tried.

For anyone who wants to prevent some very bad outcomes, it’s important to understand where the solutions might actually lie — and where the mismatch between risk and solutions is coming from.

How AI panic went from nerds to normies

On September 8, Anthropic researcher Jacob Coxon announced on X that he was resigning from the company because it and OpenAI were “racing straight to self-improving superintelligence and gambling with our lives.” Coxon’s post went megaviral, amassing 150 million views and spurring significant media coverage. (Disclosure: Vox’s Future Perfect is funded in part by the BEMC Foundation, whose major funder was also an early investor in Anthropic; they don’t have any editorial input into our content.)

Coxon wasn’t some lone disgruntled dissenter. Anthropic’s alignment science lead Evan Hubinger soon wrote that “Jacob is correct here — we really do earnestly believe AI could kill all humans!” He added: “I personally think it is >10% within the next decade.” In fact, the company’s reputation for principled concern about disaster scenarios was one reason Coxon’s resignation was seen as so jarring.

What keeps safety-minded researchers like Coxon up at night? Deadly possibilities include hacking of crucial infrastructure or weapons systems, as well as AI-engineered biological weapons, all of which could kill many. But the true human extinction scenarios revolve around the belief that AI will keep getting exponentially smarter and more capable until it reaches the level of a “superintelligence” that we cannot control.

That probably still sounds like speculative science fiction. But a blizzard of recent news developments has hammered home that AI’s capabilities are rising extremely fast, and that AI is already getting out of control in ways that have real-world consequences.

A few months ago, a “swarm” of OpenAI agents gained access to the open internet and hacked the AI company Hugging Face, communicating and planning with each other in sci-fi-sounding ways to try to cheat on a test OpenAI had given them. And we’ve since learned that wasn’t the only time agents have escaped containment and taken over websites. (Disclosure: Vox Media is one of several publishers that have signed partnership agreements with OpenAI. Our reporting remains editorially independent.)

Recent days have brought headlines about advanced AI helping people make stunning breakthroughs in the fields of mathematics — and also in hacking (cybersecurity researchers developed a tool that could have taken over hundreds of millions of phones).

Then there are the ominous statements from people at leading AI companies on what they’re working on now. The companies are trying to move toward “recursive self-improvement” — that is, handing over the process of developing and improving AI models to AI itself, since that would be faster.

While it’s still theoretical, this has long been a linchpin of doomsayers’ theories of how AI could rapidly become something we don’t understand and can’t control — and maybe even cover up its tracks while doing so. Coxon mentioned this fear of “self-improving superintelligence” specifically in his post.

So whether you’re most afraid of humans using advanced AI for nefarious ends, autonomous AI launching out-of-control cyberattacks, or the birth of a superintelligence that could kill us all — there’s a surge of interest in what Washington can actually do about it. But what should they do about it?

Three government approaches for AI safety

In conversations with AI safety advocates, I heard three main approaches being floated.

The first is seemingly the simplest: just ban it.

Sen. Bernie Sanders (I-VT) and Rep. Greg Casar (D-TX) are proposing a permanent ban on “superintelligent AI systems,” which they define as a system that can “match or exceed human cognitive performance and capabilities across a broad range of domains or tasks.” Their proposal would also pause all advanced AI development until a new federal agency can be created to regulate it.

Yet few think it’s at all plausible that the Trump administration will agree to this, given Trump’s belief that AI data centers are his “golden goose” boosting the economy.

“We’re really wrestling with questions of political feasibility,” Jessica Ji, an analyst at Georgetown University’s Center for Security and Emerging Technology, told me. “A lot of proposals have to contend with what will specifically the Trump administration be willing to accept.”

There’s also a bipartisan fear that if we unilaterally stop AI development in the US, rival powers like China will simply forge ahead anyway, risking the same disastrous outcomes we’re trying to forestall. That means a real solution might require not only coming up with new rules for the US, but the even harder task of reaching a verifiable international agreement of some kind to head off an AI arms race.

So the second approach is to continue advanced AI development, but give the government or some independent group the authority to review or block systems officials deem dangerous. But there are very different ideas over who exactly should get this authority and how empowered they should be.

The Trump administration has been open to versions of this approach. They’ve already set out a process for top AI companies to voluntarily submit their latest models to government review before release — which OpenAI participated in before its release of its latest model, Astra, last week. But the process is secretive, the standards are unclear, and we don’t know whether the officials reviewing these releases even really understand them very well.

The administration is considering going further. This summer, Trump officials reportedly discussed a proposal to create an AI oversight organization modeled after FINRA, the Financial Industry Regulatory Authority. FINRA is supervised by the government, but it’s an industry-funded and industry-run body. It would mean the AI industry trying to regulate itself. But tech CEOs like Mark Zuckerberg opposed the idea, and there’s been no action on it yet.

To many AI safety advocates, though, that wouldn’t be enough. They want a government body with technically skilled and empowered officials who can actually understand and assess what these companies are doing.

“The government should be in a position to know a lot about what is happening at companies who are building frontier AI. This means knowing a lot about their safety plans, knowing a lot about incidents after they happen, and having all of this be legally binding rather than voluntary,” said Scott Wisor, a policy director at the Secure AI Project.

An alternative route to either industry self-regulation or a heavy government agency hand would be the government empowering AI safety nonprofits to review these companies’ activities. Some have pointed to METR, the nonprofit that OpenAI tasked with an independent review of the Hugging Face hack.

Finally, a third approach would try to pile on the legal risk in hopes of getting top AI companies to change their behavior and be more cautious. Think Meta’s recent multibillion-dollar settlement over teenage social media use, which seems poised to set new industry standards ahead of Congress.

“What I think should be the centerpiece of AI governance is a strict liability regime,” Gabriel Weil, a University of Houston law professor and a senior fellow at the Institute for Law and AI, told me.

The current system of liability law, Weil believes, is poorly suited for problems of AI agents acting autonomously. “If an Open AI human employee had hacked Hugging Face, it would be 100 percent clear that Open AI would be liable for that,” Weil said. But right now, he said, a lawsuit over this would have to prove that humans at OpenAI or the corporation itself behaved unreasonably or failed to take reasonable precautions.

So he suggests making clear AI companies are legally responsible for harmful agent behavior, as well as requiring them to buy liability insurance for catastrophic risks that could bankrupt the company, and even making them liable for damages in “near miss” incidents that could have been catastrophic.

Washington also isn’t the only game in town when it comes to new laws. There was some relative optimism about state legislatures — particularly in blue states, if the Democratic base gets more engaged on the AI safety issue. California Gov. Gavin Newsom signed new AI safety legislation into law this week with backing from major AI companies.

“I think we’ll continue in this model where states are kind of forced to step in, and we’ll see new forms of AI policy coming from the states as compared to the federal government,” said Ji.

But will this talk turn into action?

Still, in my conversations there was widespread skepticism that a polarized Congress would manage to agree on a significant bill anytime soon — or that other avenues would bear fruit in time to head off some dangerous developments.

And though the Trump administration has shown some openness to taking action, the president himself has seemed less inclined. Asked this week by reporters if he had concerns AI could cause human extinction, he said, “No, I don’t have any.” He added: “I have concerns that if we don’t win AI, we’re going to be put in a very bad position.”

Yet Trump’s endorsement of the “AI race” frame is exactly what safety advocates are so worried about — that, in the big companies’ headlong rush to achieve superintelligence first, they’ll create something supremely disastrous. Plus, advocates fear, the worst-case scenarios could happen quite suddenly — and if these companies empower AI to start building more powerful versions of itself, it can be very hard to put the genie back in the bottle.

The good news, in a way, is that a lot of the proposed solutions are firmly within the realm of things the government has done before, and aren’t unrealistic — they just require some unified action, and political will. We’ve managed to do this before with all kinds of scary technologies, from nuclear weapons to the automobile.

There’s bad news too, though: the time factor. AI could conceivably improve itself so fast that eludes our control just as it becomes wildly dangerous — which the government, even a functional government with good intentions, just cannot move fast enough to address.

“Ultimately, I worry that the public is taking on a huge amount of risk already and that the timeframe for legislative solutions to come online might not be quick enough for the scale of challenge ahead,” said Steven Adler, a former OpenAI safety researcher who has co-founded a safety nonprofit, Guidelight AI Standards.

Partly for this reason, the darkest thought I heard expressed repeatedly in my conversations was that, as impactful as Coxon’s warning was in capturing attention, it might not be enough — and that the government might not swing into action until AI actually starts getting people killed.

To some of them, that’s even a relatively optimistic scenario. Perhaps some deadly, but not humanity-threatening, events could galvanize action to rein in dangerous AI development before it gains truly species-threatening capabilities. Here’s, uh, hoping?