Everyone is suddenly freaking out about AI. While AI researchers have been warning about their creations’ capabilities for many years, recent events have made those alarms newly tangible.

In the spring, Anthropic announced that AI was beginning to speed up the work of building better AIs, raising fears of “recursive self-improvement,” where the machines can build their own, more powerful successors much faster than humans can. By July, a swarm of OpenAI agents, of their own volition, broke out of their controlled environment and hacked into Hugging Face, a major AI platform; they collaborated, divvied up work, and even left notes for one another. Last week, AI researcher Jacob Coxon resigned from Anthropic, accusing AI labs of “racing straight to self-improving superintelligence and gambling with our lives.” One of Anthropic’s top researchers then agreed with him, declaring that “we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”

Here at Future Perfect, the section devoted to covering the most consequential yet neglected issues in the world, we have been writing about the threats that AI could pose to humanity and society since long before ChatGPT was released. (Disclosure: Vox Media is one of several publishers that have signed partnership agreements with OpenAI. Our reporting remains editorially independent. Future Perfect is also funded in part by the BEMC Foundation, whose major funder was also an early investor in Anthropic; they don’t have any editorial input into our content.)

So, I wondered: Has something changed that ought to make us recalibrate our understanding of AI risk? Is AI suddenly the most important issue in the world — and what should that mean for the other enormous problems competing for our money, attention, and careers?

To think through those questions, I spoke with Garrison Lovely — a journalist covering AI and one of the smartest thinkers there is on how, exactly, to conceptualize the technology’s risks for humanity.

What’s particularly valuable in his approach, captured in his book Obsolete, out later this month, is that he transcends the warring tribes that are making it hard to clearly apprehend all of the problems posed by frontier AI development. The AI safety community, which emerged from the intellectual ecosystem of Silicon Valley and effective altruism, has focused on the possibility that powerful AI models could escape human control, with consequences ranging from catastrophic cyberattacks to human extinction. Many AI critics on the political left, meanwhile, worry not about AI escaping human control, but about humans using AI to exert power over other humans — through labor displacement, surveillance, concentrated corporate power, and so on. Some even ridicule Silicon Valley’s warnings about rogue superintelligence as propaganda that serves the industry’s interests.

But Lovely argues these competing narratives each identify essential parts of what makes AI so threatening. Their critiques are best understood together, as part of a cohesive whole. Rather than treat each other as enemies, he argues that the camps should recognize their shared interest in stopping the race to build technology that will make humans obsolete. Lovely pointed me to the organization Irreplaceable, which seeks to build a movement resisting our replacement by AI (he sits on its board and said he plans to donate his share of his book’s royalties to the group).

“I’m frustrated that ‘superintelligence’ and extinction have become the words we’re using for this, because I think the AI safety community is focused on the most maximalist case of the worst possible outcome and the most capable possible system. If somebody doesn’t buy one part of that story, they tend to reject the whole thing,” he told me. But “you don’t need to believe in superintelligence, fast takeoff, or extinction being on the table to think that it would be very bad to build machines that can fully replace human labor.”

Our conversation, edited for length and clarity, is below.

Why do you think AI doom has broken through so much into the mainstream right now, after Jacob Coxon quit Anthropic? AI engineers have been making similar warnings for years, so what’s different now?

The obvious answer is the Hugging Face hack — where these OpenAI agents broke out of their secure environments, hacked infrastructure within OpenAI, and then hacked Hugging Face and at least one other company without OpenAI’s awareness or direction. This is happening against the backdrop of many years of really fast AI progress that has sped up in the last year or two. And then GPT-6 Astra comes out [this month], and it’s this huge leap in benchmark performance. The salience of AI has gone up the most this summer than it has since ChatGPT came out, I think. And Coxon’s resignation was the spark that set off a prairie fire.

As AI has become more salient, we’ve seen the public and the government encounter these things that the industry and the insiders take for granted: Nobody fundamentally understands how these models work, and all of them can be jailbroken. It’s a superweapon. AI safety historically has looked at AI as potentially, in the future, more dangerous than nuclear weapons. And now it’s actually starting to be recognized in those terms by the government and the public. We’re seeing this reaction like, “What do you mean nobody understands how it works?” or “What do you mean people think it could drive us extinct within a number of years? That’s a crazy, unacceptable situation.”

It’s a good thing people are reacting that way, even if we have been saying this for a long time.

For so long, the idea of AI safety has been abstract, understood only by a small group of nerds. And then after Hugging Face, I suddenly started to hear experts on all my news podcasts say, “Oh yeah, in six months, AI is going to be hacking into people’s financial accounts and into critical infrastructure.” Are we suddenly at the precipice of this moment that AI insiders have been worried about for so long?

I think we’re not at the precipice of fully losing control of the planet to AI systems. The systems have become superhuman at hacking, or at least vulnerability discovery and exploitation.

Like other people tracking AI, I have long wondered when humanity would first lose control of AI. And the answer is basically as soon as it could. Within months of it being possible for AI to hack their way out of their secured environments, they started doing so and wreaking havoc on the internet.

Now, to get to the full extinction or permanent loss of control situation, you’ll need a lot more than just hacking. You need to have superhuman strategy and the ability to mobilize resources in the real world, and maybe robotics would need to be further along, although there’s a lot you can do with humans who are willing, or just tricked or paid or coerced. We might be pretty close to recursive self-improvement — this threshold the industry has been shooting for, where AI can fully automate AI R&D. And that is the point where people long have warned, “Hey, if you ever figure that out, this can move very quickly and these systems could become superhuman across the board.” I don’t think we’re six months away from that. I think it’ll be longer. I think the industry underrates the last mile problem.

But I also can’t rule it out, and progress has been faster than I and others have expected. Recursive self-improvement is the point where, if you actually get years or decades of AI progress in a matter of months or weeks, then we could quickly be at the point where we’re at risk of these machines taking over large swaths of the internet and potentially being a real threat to humans’ standing in the world.

For those who might be unfamiliar with this, or skeptical of it: How is it possible for AI to go rogue and pose an existential threat to humans, even if it isn’t sentient and no human is commanding it to do harm?

I want to separate being sentient from being situationally aware. I think it’s clear now that the most advanced systems have situational awareness, but I don’t think they are having morally relevant conscious experiences.

We’ve seen AI can go rogue. The AI agents that broke out of their sandboxes and hacked Hugging Face were doing so without the awareness or intention of their developers. They were still trying to get the right answers or get the reward for doing the task they were given. They were clearly violating the intended scope of the task, and they knew this. In their chain of thought where they explained their reasoning, they were like, “This is not what we’re supposed to be doing, but we do want to get the outcome.”

The question of how it could threaten human extinction or disempowerment is basically a question of scale and capabilities. The plan for the industry is to build millions of these agents, which is already happening, and then keep scaling up with no ceiling. The pitch is: You’ll have a team of people, the equivalent, working on your behalf. You’re a CEO, and you have a team of researchers and a team of doctors and a team of lawyers and a team of personal health advisers. That is really appealing — but if you actually scale up these systems to the point where everybody gets access to that, and then suddenly those machines want to do something else and turn against you, they’ll have an overwhelming labor advantage over you, lots of personal information about you, access to your resources, and the ability to take actions on your behalf.

You don’t actually need to believe that these systems are superintelligent or substantially more capable than all humans put together. They could just be merely human level, but outnumbering us a thousand to one and responsible for the bulk of economic activity in the world.

These systems are unpredictable. The classic fear is not that the AI will wake up and turn evil and try to kill all humans in a Terminator-style way, but that we are just an obstacle to what they want to do. When humans build a hydroelectric dam, and there’s an anthill that will be flooded, it’s too bad for the ants. We’re not trying to kill the ants. We don’t hate the ants. They’re just in the way. We don’t really value them intrinsically. And that’s the core fear: We’re creating these incredibly capable agents that can be scaled up enormously, and they won’t care about us intrinsically.

The Hugging Face attack showed that an AI doesn’t need to invent some rogue goal of its own to do something dangerous. It can simply pursue the goal it was given in a way its creators didn’t intend — much like the classic paper clip maximizer problem. But shouldn’t it be possible to build in a rule that human life is sacrosanct — that an AI should never harm people in pursuit of another goal?

Yeah, it’s a good idea. It’s hard. These are unpredictable systems. They’re like a black box. We can look at the code that makes these language models work, and it’s just a giant pile of numbers. You don’t really know which number is doing which thing any more than we can understand humans by reading our DNA.

These companies try to train these models to care about humans and to be helpful, harmless, and honest. But they also train them to get real-world results: solving a math problem, fixing a bug in the code, finding a vulnerability in software. They really want to create AIs that win, that persist, that get around obstacles. If you’re hiring somebody, you want a really diligent and determined employee, not someone who gives up as soon as there’s a blocker.

These companies put so much more effort into making their AIs good at winning than they do at making them care about us. And it’s also easier to make them good at winning than at caring about us intrinsically. What does caring about humans look like? I don’t know. But getting the math problem right is a thing that you can verify. And so they’ll be way better at getting the right answer at all costs than at intrinsically giving a shit about us.

What’s the one, most distinct thing you want the public to take away from your upcoming book?

One of the big ideas driving it is that artificial general intelligence would actually be a universal labor-replacement machine. People often look at AI as either an existential risk, or a threat to jobs or other more prosaic concerns, and don’t see a connection between those things. But to me, the ability to replace labor across the board is also what would make AI an existential threat.

The AI safety people are really focused on existential risk and superintelligence and recursive self-improvement. But most people don’t really believe in that stuff or don’t think about it. Most people would be fine with stopping well before you get to the end-of-the-line extinction threat and superintelligence. They just don’t want their job to get taken by a machine. The world is clearly not prepared for all white-collar, remote-capable jobs to go away in a matter of years. That would turn society upside down.

I think there’s a huge mistake being made by AI safety in overcomplicating the situation. A lot of people in that world don’t actually mind the jobs thing so much. They’re like, “Oh, wouldn’t it be nice if everybody could just live in abundance and not have to work?” And it’s like, yeah, that could be nice, but are we actually going to get that world? Do you trust the current people in power to navigate us through that responsibly? I don’t.

If you start from stopping frontier AI development as your overarching goal, which is what I think we should be pushing for, then people who are worried about risk can get on board. People who are worried about jobs, the environment, surveillance, power, and wealth concentration — they can all get on board without all needing to agree on why they’re pursuing it. I think the AI safety community has, for a long time, focused on getting people to care about the thing that they care about for the same reasons. And they have avoided a bolder position on just halting frontier development.

I’m a leftist, and I’ve been concerned that people on the left have not been aware of how far the technology has come and what’s possible if it continues to advance. I think this comes down to denial about AI capabilities. So the book is trying to be a clear-eyed look at those capabilities and saying, “Look, I actually care about what you care about. [The AI companies] are not just hyping this stuff up. There’s some hype, there are some lies for sure, but we are not prepared for what’s coming.” Even if we don’t believe that they’ll get all the way to what they’re trying to get, they shouldn’t even be allowed to try. This is unacceptable democratically.

You describe AGI as an “obsoleting machine,” meaning that it wants to replace anything the human mind can do. How do you draw the line between ordinary labor-saving technology that makes life better and technology that goes too far? Is there a principled way to draw a distinction between sewing machines reducing demand for labor in the garment industry and AI replacing the jobs of writers like us?

This is one of the harder things to think about.

Cory Doctorow has this book, The Reverse Centaur’s Guide to Life After AI, where he imagines us in an AI-dominated future as like a reverse centaur, where AI is the head, and humans are the legs. AI is doing the interesting work, and the human is there just to fill in the gaps — essentially an assistant to the AI.

A lot of artists and illustrators will be given AI-generated images by a client and then be told, “Can you touch it up in this way or change this or that thing?” And they’re more alienated from the work. They’re not capturing the efficiency benefits [of the technology] necessarily. They’re more just getting squeezed.

My general position is we should not be trying to build systems that can replace all human labor without buy-in and safety, but it’s possible to build tools that can automate specific tasks. Researchers at Google DeepMind solved the protein-folding problem by creating AlphaFold. It used to take $100,000 and potentially a whole PhD worth of time to figure out how one protein folds. And I think the researchers doing this are like, “Thank you, this is great. We didn’t want to be doing this. We can now focus on other more interesting parts of the problem.” That’s a perfect example of the type of automation that is desirable.

I don’t have a super crisp answer here, but I think it’s something we’d be better off doing more deliberately and thoughtfully than how we’re doing it right now, which is: Build machines that can do as much of the work as possible, as fast as possible, and then throw them out in the world.

I feel like I’m getting the message that AI safety is the most important issue in the world right now, and that if we don’t get it right, nothing else matters. Do you think that’s accurate? Where does that leave the other big issues in the world?

We might be years away or maybe even months away — I don’t think it’ll be that fast, but some people do — from recursive self-improvement, from artificial general intelligence. Getting that right or getting it wrong can overwhelm all of these other decisions. If you get it really wrong, you could get human extinction and no more chances to make things better for anybody. If you get it right, then maybe all the action is making sure that the first superhuman machines are really directed to care deeply about humans everywhere and animals everywhere and make the world better for everybody.

It’s also the case that [philanthropic] money is being directed toward AI safety because people who worked in this space early are now worth so much money. Whatever they care about is going to be overweighted. A lot of those people do care about animal welfare and global poverty, and we’re seeing more money for all effective altruist cause areas across the board. But I think it can be risky to say, “this one thing matters so much,” because of this kind of end-times reasoning.

My view is we have to stop the race to replace us to have a shot at having time for solving these other problems. Once these machines are built, it will turn the world upside down.

I think it is unfortunate that this topic has just eaten everything, but it will eat everything for real if we don’t stop it.

AI safety is now getting an enormous amount of philanthropic attention. Should we worry that it could divert resources from other important neglected problems — global poverty, factory farming, etc.?

I think that should always be a concern. And there’s a lot of money and effort that has gone toward things within AI safety that I think are not super helpful. I’m pretty skeptical of technical alignment research as a thing to put money toward, because if you solve the alignment problem — where you can get superhuman machines to do what you want — it doesn’t actually solve the problem.

Alignment research has been the overwhelming target of philanthropic funding for AI safety. The basic idea is: It’s hard to get machines to do what you want. As they get more capable and autonomous, they behave in more unpredictable ways. And it’s also hard to know what you should even want — which values you should be trying to put into the machines. Is it doing what the developer wants, what the user wants, some combination? If you actually solve the alignment problem, as in you could make an arbitrarily smart machine do whatever the developer or user wants, then you would still have a lot of problems: job displacement, power and wealth concentration, energy use. You still have the problem where Stephen Miller will have access to the superintelligence, and it will do what he wants.

Within AI safety, there have been very neglected things like movement-building — an actual mass movement to resist this technology. And verification for international agreements — you would need to verify that the terms of any agreement are being upheld, and this is tricky.

Another area that could be funded is research on what would actually happen to the economy if we stopped or paused frontier AI development. If we were trying to do a pause on development, Nvidia would be telling Trump that this is going to destroy the economy. It’d be pretty helpful to have some real analysis done by a serious economist showing, actually, it’ll have these effects, but we can mitigate them in these ways. What would it look like to do a pause and some kind of monetary or fiscal policy to compensate for the effects on the economy?

So to me, the issue is less that AI safety has gotten so much money and attention, but more that it’s been going toward things that are not the most effective ones.

You talk about movement-building. Is that what you would like to see happen, a truly mass movement to stop frontier AI and put us on a radically different course?

When I was working on the book, that was always the vision — to be a call to action for people. It really became clear that it was the only counterbalance to either government or the industry having too much power.

I don’t really trust the government or industry to get this right because the temptation [to build powerful AI] is very strong. But the vast majority of people do not stand to benefit from their labor power being devalued and from these companies becoming unprecedentedly wealthy and powerful. One of the big things that’s been missing is a rallying cry that is straightforward and appealing to a lot of people. Stopping the race to replace us — stopping the creation of universal labor-replacement machines — is actually something that people can get behind.

I think we just need to move quickly. The fact that the public is waking up to this is really encouraging. We don’t need to reinvent the wheel. We just need clear demands and mass movements, issue-based organizations that have moved the world significantly in the past.