I found myself fielding a lot of questions from folks who don’t usually talk much about AI this week. They were mostly along the lines of “do you think AI is going to kill us all?” or “how worried should I be about this human extinction stuff?”
Motivating this wave of existential anxiety was the highly dramatic resignation of now-former Anthropic employee Jacob Coxon, who quit his job while publicly sharing his concerns about safety practices at the leading AI companies. More specifically, he declared that almost everyone at OpenAI and Anthropic, the leading AI labs, believes that there’s a decent chance the AI systems they’re producing could “kill us all” by the end of the decade, and the companies aren’t “acting responsibly.”
Other AI safety researchers, including those still employed at Anthropic and OpenAI, all confirmed that yes, Coxon is correct, they’re all worried that their company’s products might prove fatal for the entire human race as well.
Two more AI employees quit and joined the chorus. Now, the warning shot has been covered everywhere from the Wall Street Journal to Fox News to WIRED to the Guardian.
I understand why all this sounds scary, but my response to those who have asked, has generally been: No, you don’t need to worry that AI will kill go rogue and kill us all. But that does not mean we shouldn’t worry, period. Far from it. The companies building AI products are now among the wealthiest and most powerful entities in the world. And they very much are building enormous engines of mass surveillance, extraction, and automation. They can and have allowed their AI systems to carry out felony hacks of other companies, provided the government with autonomously generated kill lists, and sold enterprise technology to firms around the world that is explicitly intended to replace jobs. But “AI” has done these things not because they have all-powerful minds of their own but, ultimately, because people have programmed, calibrated, and directed them to.
In other words, it’s not AI we need to worry the most about. It’s the people and corporations working to profit from and gain power by deploying it. The technology is sophisticated, sure, and, especially in cybersecurity contexts, potentially disruptive to plenty of existing systems. But if we want to stop this predatory, profiteering, and extractive form of AI—and it’s worth noting that at this point, most people do—we absolutely can.
A quick note/plea: I can only do all of this with the support of readers like you, so if you find value in independent, critical tech journalism and analysis, please consider becoming a paid subscriber. I want to keep as much of this un-paywalled as absolutely possible, and I need your support to do so. Many thanks to all who continue to chip in, it’s very much appreciated. Onward.
First, some context: AI industry folks like Coxon have been discussing the prospect that AI will destroy humanity for years. Even before the beginning of the AI boom, many working in the industry, from the executive staff to the rank and file researchers, have obsessed over ‘x-risk’, or the possibility that AI will become sentient or semi-sentient and move one way or another to exterminate humanity.
This is ostensibly why Elon Musk bankrolled OpenAI in the first place, and is the grounds for his lawsuit alleging OpenAI betrayed its charter by abandoning its nonprofit status to become a commercial enterprise. Many AI engineers and startup founders have calculated their own personal p(doom), or the probability that the tech product they are working on will initiate an actual humanity-extinguishing apocalypse. 10-25% p(doom)s are pretty common. (I wrote about a long weekend I spent among these folks at one of their top conferences here, to give you an idea of their ideas and belief systems.)
Sam Altman himself drove the narrative of the imminent rise of a super-powerful AI, or artificial general intelligence, to prominence by declaring that the technology he was building posed an existential risk to humanity.
This idea is in other words baked into the DNA of modern AI companies and into the products they engineer. It’s also why the industry is routinely accused, by myself and others, of “doom marketing,” ie, using such proclamations and framings to impart the sheer power of the technology behind your product onto investors, partners, consumers, and the state. Some ambient understanding of this tendency may have helped prevent the press from repeating the claim every time one of the x-risk folks made it over the years, including when they quit their jobs at AI companies while doing so.
Yet Coxon’s AI x-risk alarm has broken containment, as they say, and has exposed the idea to the mainstream media once again. This has happened, I think, for a number of reasons. First, when AI CEOs were going around saying AI might kill everyone in 2023, their companies were well-capitalized, but not nearly to the extent they are today. They were more easily dismissed as eccentric geeks of the sort that Silicon Valley is supposed to produce. Now, not only do their products have much greater market saturation, but people are more aware of a) the often unethical ways the executives and companies comport themselves b) the other adverse impacts of AI that have already arrived, and c) the physical cost of building these systems, particularly when it comes to data centers. Finally, toss in the constant stream of headlines in 2026 about AI models going rogue or potentially upending the world’s infrastructure.
As a result, general sentiment was already overwhelmingly skeptical of AI and its overlords, so the table was pretty well set for someone to come along and say ‘just a reminder, the companies that are building the product that promises to take your job and leave a data center in your neighborhood also think that it might exterminate the human race’ and get a strong reaction from the media and the public.
The question is: Should we understand this stuff as hype, strategically fanned to feed already swelling valuations? Is it doom marketing? Genuine cause for alarm that the infrastructure that undergirds modern life is facing a novel threat vector? Or reason to be sincerely worried that AI companies are amassing too much power, and that their products are genuinely dangerous?
The answer, I’m afraid, is “yes.”
Let me try to untangle all of this a bit, because arguments over how to navigate this moment are extremely heated right now, as they are essentially arguments over how to approach the politics of AI at a moment where there may be a unique opening for action. Those sympathetic to AI alignment concerns and x risk arguments—which increasingly includes a faction on the left who want to shut down AI over safety concerns—argue that people who dismiss AI as all hype have their heads in the sand. Meanwhile, critics of the x-risk narrative—which I might note includes a cadre of investors and tech folks on the right, too—argue that it is hype and can be used to further inflate valuations or for pursuing regulatory capture.
The challenge is that many feel that dismissing AI as just hype or purely marketing also diminishes the urgency of taking on the AI industry as the novel and threat to labor, civil society, and the public sphere it is; others that by accepting the industry’s own terms, we imbue it with undue power and enable it.
But let’s be clear: Declarations that AI is so powerful it might end the world are a kind of marketing, and anyone who says otherwise is just as off base as those who say that’s all that it is. If you don’t think that these sub-trillion dollar companies have higher-ups who noticed that when their founders talked about super-powerful dangerous AI they got spikes in media interest and phone calls from investors, or that those founders didn’t notice that themselves, and lean into certain narratives to advantage their products, then I don’t know what to tell you. Not all marketing is a Pepsi ad campaign with a celebrity spokesperson.
Yet confronting the AI industry should certainly be considered an urgent project. Do I share the x-riskers’ fear that out-of-control, super-intelligent AI agents might end the world? I do not. I’ve read a good deal of AI extinction literature over the last couple of years and I have not come across a credible, step-by-step documentation of how exactly AI might move from self-recursively improving AI to killing every single human on the planet. Most of what I’ve read seems predicated on a sort of reverse Ian Malcolmian logic: Eventually, it becomes smart enough and AI, uh, finds a way (to exterminate humanity). Are the AI companies also overstating and dramatizing what their products can do? Yes, pretty much all the time. They’re in a knife fight for market share and are trying to win customers and government contracts before their investors come knocking.
And yet. They have concentrated enormous amounts of compute, have extensive data collection and surveillance capacities, have inserted themselves into the state in numerous capacities, including military kill chains, and they are being run by egomaniacal billionaires who are rushing to monopolize an entire new market. The way I think about their most recent frightening-seeming exploits is not as a “swarm of rogue AI agents building a new civilization” or whatever, but as corporations with a hitherto-unimaginable amount of compute at their disposal recklessly experimenting with automation programs, completely unafraid of any repercussions because we have been unwilling or unable to impose any on them. So I think we should welcome any newfound imperative to hold them accountable.
I also think that ultimately, the AI critics, Pause AI folks, leftists, doomers, effective altruists, and, honestly, most of the public, want a lot of the same things: legitimate checks and limits on AI companies’ power, actual transparency into what the companies are doing, and for more of AI to be under public control. (Ultimately, a lot of what animates the x-risk fears are rooted in just that: a fear of losing control.)
How we move towards all this, exactly, is the question of the day, and while there’s much to be argued over in terms of building a viable political platform to challenge the AI industry’s power, I do want to cast a vote against one particular approach.
Not to pick on Chris here, as there are a lot of folks out there proposing stuff like this, and the policy ideas actually stem from AI safety groups. But not only would this be challenging to land politically (I can already hear the ‘China will beat us to AGI’ calls) and all but unworkable in practice, this approach is a) far too limiting, b) fails to address the current harms caused by AI and c) would likely only wind up serving Anthropic and OpenAI; it’s what regulatory capture looks like in action.
Full model transparency is the floor for what should be on offer here. If these companies believe their products will kill us, the very least we deserve is to know how. It’s in the public interest. I would also argue we might get a lot of the way to stopping any currently envisioned robot apocalypse simply by forcing AI companies to assume liability, and to submit to the full brunt of the law. If Sam Altman were liable to be charged with a felony for approving the hack on Hugging Face1, if Google was forced to pay defamation fines each time it published a false and injurious AI Overviews, these companies might suddenly find cause to restrain their models’ output, and we might hear less about rogue AI. We would need to introduce new frameworks and available tools for tracking automated lawbreaking, to be sure, but I’d wager that little would frighten the AI companies more than actually being held accountable for what their products enable. To that end, Illinois has passed an AI liability law, which the industry fought tooth and nail, and Florida is considering one.
The next steps would be towards moving AI under public control; again, it’s only reasonable, since everyone, not just AI insiders, should get a say over how a technology whose developers say might kill us is developed and deployed. Dismantling the whole industrial project as currently structured should be on the table, too, pending democratic input. But that’s for another post.
For now, suffice to say that while I won’t be losing any sleep about an AI-enabled extinction event, Mr. Coxon and co. have given us an opportunity to underscore the urgency of the clear and present damage AI companies are doing, the extreme power they wield and the extent to which they hold themselves unaccountable, and the abundance of political capital there is out there for taking them on.
Purely speculative! I do not know if he did or not.
“I also think that ultimately, the AI critics, Pause AI folks, leftists, doomers, effective altruists, and, honestly, most of the public, want a lot of the same things…”
That’s a really important point that easily gets lost in the day-to-day debates (some civil, some not) between those groups. The interests of “we the people” have a lot more in common when contrasted against the interests of the tech oligarchs.