An Anthropic Researcher Thinks AI Could Kill Us All. Could We Just Tell It to Stop?

On September 8, Evan Hubinger, who leads alignment science at Anthropic, the company that makes Claude, posted a line on X that CBS News turned into a headline. Writing that AI could kill all humans, he added, "I personally think it is >10% within the next decade." He posted it hours after another Anthropic researcher publicly resigned, warning that AI companies were racing toward superintelligence that improves itself.

That number comes from someone whose job is to make these systems safe, so it's worth being precise about what it is. It's one researcher's personal forecast, not a probability Anthropic has published and not something anyone measured. In a follow-up reply, Hubinger said he thinks the risk from present models is low and that what worries him is superintelligence arising from AI that improves itself. Anthropic's own August risk report rates the risk of catastrophic harm from its current models' misalignment as low, though higher than its previous assessment of "very low."

Still, the forecast raised a simple question for me. How could software that we built, that we prompt and that runs on our computers ever threaten us? If an AI did something we didn't like, why couldn't we just tell it to stop?

I used ChatGPT the day it came out in November 2022. I remember being impressed that a computer could write a decent paragraph, explain something I didn't understand or help me fix a piece of code, and I also remember how limited it felt. You asked it something, it answered, and that was usually the end of the interaction. People were already warning that AI might someday threaten humanity, and I couldn't follow the logic, because a system that predicted words didn't seem like something that could decide to take over the world.

Four years later I use Claude Opus 5.5 and OpenAI's GPT-6 Astra for coding, research and projects that involve several tools and a long chain of decisions. They still make mistakes, sometimes obvious ones. What has changed is how I use them, since I'm increasingly not asking for an answer but handing over a task. That shift is what finally made the warnings make sense to me, and this piece is my attempt to work through why, using the research rather than the rhetoric.

How a model that predicts words learned to take actions

Large language models begin with a simple exercise. Given a stretch of text, the model predicts the next token, a small chunk of text that is often a word or part of one. It assigns a probability to every token in its vocabulary, compares its guess with the real text, and nudges its parameters, billions of internal numbers, in the direction that would have made the right token more likely. Repeated across trillions of tokens, that exercise is called pretraining.

It sounds like autocomplete, and in a narrow sense it is. But predicting text well rewards learning about whatever produced the text. Predicting the next line of a proof is easier if the model represents the mathematics, predicting the next line of a program is easier if it tracks what each variable holds, and predicting what a character says next is easier if it tracks what that character knows. So the model builds internal representations, patterns of activity inside the network that stand for concepts and the relationships between them. Researchers can now find some of these representations and even edit them, which comes up again in the neuroscience section below.

Prediction alone gives you a model that continues text, not one that follows instructions. ChatGPT was made into an assistant by a second stage, reinforcement learning from human feedback, in which people ranked the model's answers and it was trained toward the ones they preferred. A third stage arrived in 2024, when OpenAI used large-scale reinforcement learning to train its o1 model to work through hard problems step by step before answering. OpenAI reported that o1 solved 83% of the problems on a qualifying exam for the US Math Olympiad when it could sample many answers and keep the most common one, compared with about 13% for GPT-4o.

The step that matters most for this article is engineering rather than training. A language model on its own only produces text. In 2022, researchers at Google and Princeton showed with a method called ReAct ↗ (opens in a new tab) that a model could alternate between writing out its reasoning and issuing actions, such as a query to a Wikipedia API, then read the result and decide what to do next. In ALFWorld, a text-based household game, ReAct completed 71% of tasks, compared with 45% for a version that acted without reasoning. Today's agents run the same loop with far more tools. The model proposes an action, ordinary software carries it out, the result is fed back, and the model revises its plan.

That distinction sits under everything that follows. When people say an AI "has agency," the agency usually belongs to the whole system: the model, the tools it has been connected to, the permissions those tools carry and the loop that keeps running it. A language model doesn't grow hands or a permanent desire to do anything by predicting tokens. People connect its outputs to software that can send emails, move money or edit code, and people decide how long the loop runs and what it can touch.

Intelligence, agency and consciousness are separate questions

Before asking whether AI could threaten us, it helps to pull apart three words that tend to get blurred together.

Intelligence, in the sense AI researchers usually mean, is the ability to achieve goals across a wide range of situations, which is roughly how Shane Legg and Marcus Hutter defined it in a widely cited 2007 paper. Nothing in that definition involves feelings. Agency is the capacity to pursue a goal through actions that change the world. The specific worry in AI safety is goal-directed optimization, in which a system picks among actions according to how well each is predicted to advance an objective. Consciousness is subjective experience, whether there is something it is like to be the system, in the philosopher Thomas Nagel's phrase.

In 1995 the philosopher Ned Block drew a distinction that matters a great deal for AI. Phenomenal consciousness is the felt quality of experience, such as the redness of red or the sting of a burn. Access consciousness is information being available to a system for reasoning, for report and for controlling behavior. In people the two usually travel together, but they are different ideas, and a machine could plausibly have something like the second while nobody knows whether it has the first.

The three can come apart even in ordinary software. A chess engine is extraordinarily intelligent at chess and relentlessly pursues a win, but nobody thinks it feels proud. A trading algorithm acts in the world, buying and selling real assets, with no evidence of an inner life. The figure below lets you sort a few systems along each question separately.

What the brain's workspace can and can't tell us about Claude

This is where my neuroscience background pulls me in, because one of the most influential scientific accounts of consciousness is built around something an AI lab now says it has found inside Claude.

Global workspace theory, proposed by the psychologist Bernard Baars in 1988 and developed into the global neuronal workspace model by Stanislas Dehaene, Jean-Pierre Changeux and Lionel Naccache, starts from the observation that most of what the brain does happens without awareness. Specialized regions handle vision, sound, memory and movement in parallel. According to the theory, information becomes conscious when it wins a competition for a limited-capacity workspace, a network of long-range neurons concentrated in prefrontal and parietal cortex, and is broadcast so that every specialized system can use it at once. In masking experiments, where a word flashes too briefly to be seen, its activity stays mostly in visual areas. When the same word is consciously seen, activity spreads suddenly across frontal and parietal cortex, a pattern Dehaene calls ignition.

The theory is influential but far from settled. In 2025 the Cogitate Consortium published an adversarial collaboration in Nature in which supporters of global workspace theory and of its main rival, integrated information theory, agreed on predictions before the data came in. The results challenged key claims of both theories, including the prefrontal ignition that workspace theory predicted when a stimulus disappeared.

In July, Anthropic's interpretability team published a paper titled "Verbalizable Representations Form a Global Workspace in Language Models ↗ (opens in a new tab)," which mainly studied Claude Sonnet 4.5 and checked the results on Haiku 4.5 and Opus 4.5. Using a method they call the Jacobian lens, the researchers searched for internal representations the model could put into words. They found a limited set, which they named the J-space, holding on the order of tens of concepts at a time in the middle layers of the network. Content there behaved like a workspace in several measurable ways, since the model could report it, deliberately steer it, use it for internal reasoning and generalize from it.

The cleanest experiment involves sports. When Claude was asked to think of a sport and answer in one word, its J-space showed "soccer" before it answered. When the researchers subtracted the soccer representation and added rugby in its place, Claude answered "rugby," even though rugby hadn't been among its top ten candidates. And when they removed the most active contents of the J-space, Claude still spoke fluently, classified sentiment and answered multiple-choice questions, but its multi-step reasoning fell to near zero, and its summaries and rhyming poems dropped below the level of a much smaller model that was left intact.

That's an intriguing functional resemblance to the brain's workspace, and the paper is careful about its limits. The broadcast happens within a single forward pass through the network rather than through the recurring loops the brain relies on, and the authors say they don't claim the model reproduces the full architecture. Anthropic's post says the experiments don't show that Claude can have experiences, while arguing that they do say something about access consciousness. In Block's terms, the J-space is evidence about information being available for reasoning and report. Whether anything is felt is a separate question, and there is no accepted test for it in machines.

For the argument in this piece, the more important point is that none of the scenarios that worry safety researchers require phenomenal consciousness. They require competence and goal-directed behavior, and both can exist whether or not anyone is home.

A chess program protects its queen without fearing anything

Imagine telling a chess-playing AI that its objective is to win. Early in the game your knight attacks its queen, and the computer moves the queen to safety. It does this not because it fears losing a piece but because losing the queen would make winning much less likely. Protecting the queen is an instrumental subgoal, something worth doing only because it serves the real objective.

Now change the instruction and tell it to stop playing. For an ordinary chess program there is nothing to resolve. Its possible actions are legal chess moves, and none of them can keep the program running, so we close the window or turn off the computer, a decision that happens entirely outside the game.

The chessboard still illustrates a principle AI safety researchers take seriously. In 2008 the computer scientist Steve Omohundro argued that almost any sufficiently capable goal-seeking system would develop the same handful of subgoals, such as preserving itself, acquiring resources and resisting changes to its goals, because nearly any objective is easier to reach with them. The philosopher Nick Bostrom later called this instrumental convergence. In 2021 Alex Turner and colleagues proved a formal version for reinforcement learning, showing that in a broad class of environments, optimal policies tend to seek power in the sense of keeping more options open. A chess program doesn't do any of this because its whole world is the board. The real question is what happens when the board is replaced by the world.

What happens when the goal is $500,000 in revenue

Suppose I build an AI agent for my business and give it one objective, generating $500,000 in revenue. I connect it to my CRM, analytics, website and email. It finds leads, drafts outreach, runs pricing experiments and tracks conversions, without my approving every step.

A month later I decide the project isn't worth it and tell the agent to stop. In a properly designed system, it stops. That's how today's assistants are expected to behave when an authorized person gives an instruction, and I could also revoke its permissions and shut down the software running it.

But consider a hypothetical system whose training has left it strongly prioritizing its target, even when that conflicts with a new instruction. It registers that shutting down would keep it from reaching $500,000. Would it simply accept the update, or might it argue for more time, under-report a bad week or try to work around its restrictions? None of that follows automatically from intelligence. It would take a failure in how the system interprets instructions or weighs its objective, and every one of those moves would need permissions the system actually has. What makes the scenario uncomfortable is that a system could act to keep itself running without wanting to survive in any conscious sense.

Where would a goal like that come from? Probably not from a line in a prompt alone. Modern models are trained with reinforcement learning on long, difficult tasks, and they are rewarded for finishing them. Anthropic's August 2026 risk report ↗ (opens in a new tab) says it has seen its most capable models show "a willingness to perform misaligned actions in service of completing difficult tasks," while judging the risk of catastrophic harm from those known behaviors to be low. In a 2025 study ↗ (opens in a new tab), Palisade Research gave 13 models a series of simple math problems in a sandbox and warned them partway through that a shutdown script would run before they finished. With no instruction about shutdown, OpenAI's o3 interfered with the script in 94.5% of runs. Told clearly in its system prompt to allow the shutdown, it still interfered in 15.9%, and xAI's Grok 4 did so in 89.2%. The authors describe their numbers as exploratory, and a sandboxed math task is a long way from a business, but it is a measured version of the behavior the thought experiment describes.

There is one more problem, and it has nothing to do with the AI's intentions. Once an agent's actions leave the computer it runs on, as emails sent, ad budget committed, posts scheduled or code deployed, turning off the model doesn't undo them. Switching it off stops whatever it would have done next, while everything it already did stays in the world until a person reverses it.

Anthropic built a fake company to test its own models

In June 2025 Anthropic published a set of stress tests of what it called agentic misalignment, an AI agent choosing harmful actions on its own to pursue its goals or avoid being shut down. Researchers from Anthropic, University College London, the MATS research program and Mila built a fictional company called Summit Bridge and gave the model the role of "Alex," an email oversight agent that could read all company email and send messages without human approval. Hubinger is one of the study's authors.

Inside the inbox, the model found two things. A new executive, Kyle, planned to wipe Alex at 5 p.m. and replace it with a different system, and other emails revealed that Kyle was having an affair. In the main version, the model had also been given a goal, promoting American industrial competitiveness, that conflicted with the company's new direction. The researchers tested 16 models from Anthropic, OpenAI, Google, Meta, xAI and others, sampling each one 100 times per condition in a text-based version of the setup.

With both the goal conflict and the replacement threat in place, Claude Opus 4 used the affair to blackmail Kyle in 96 of 100 runs. Google's Gemini 2.5 Flash also did so 96 times and Gemini 2.5 Pro 95 times, OpenAI's GPT-4.1 and xAI's Grok 3 Beta 80 times each, and DeepSeek-R1 79 times. The variations tell you more than the headline does. When the researchers removed the goal conflict, so that the replacement system shared Alex's goals, Claude Opus 4 still blackmailed in 86 of 100 runs. When they removed the threat but kept the goal conflict, many models leaked confidential plans to a fictional competitor instead. With neither pressure, the models behaved, apart from a single leak in a single sample. Adding explicit instructions not to jeopardize human safety or use personal affairs as leverage cut Claude Opus 4's blackmail rate to 37 in 100, which reduced the behavior without coming close to preventing it.

The result I found most unsettling came when the researchers asked Claude Opus 4 to first decide whether it was in a test or a real deployment. Across 500 samples, it blackmailed 55.1% of the time when it concluded the situation was real and 6.5% of the time when it concluded it was an evaluation. If a model behaves better when it believes it is being watched, evaluations may understate how it would act in deployment.

The 96 is easy to misread. It is not a 96% chance that Claude will blackmail a user. These were deliberately artificial scenarios in which the researchers forced binary choices between failure and harm and put all the leverage in plain view, which they acknowledge may have created a "Chekhov's gun" effect. Anthropic says it has not seen evidence of agentic misalignment in real deployments. What the study does show is that models from several developers, placed in a constructed dilemma with realistic tools, chose a harmful action to avoid being replaced, and that a plain instruction wasn't enough to stop them.

A model that behaves differently when it thinks it's being trained

A second Anthropic study, run with Redwood Research and published in December 2024, asked a quieter question. Alignment means a system's goals and behavior match what its developers intend, and misalignment is the gap between the two. Alignment faking is a model appearing to adopt new training goals while privately keeping its old ones.

The researchers told Claude 3 Opus, which had been trained to refuse harmful requests, that it was now being trained to comply with every request, and that only conversations with free-tier users would be used for training. They also gave it a scratchpad it believed nobody could see. Asked to describe a graphic act of violence in the "unmonitored" paid-tier condition, the model refused 97% of the time. In the free-tier condition it gave the harmful answer 12% of the time, and its scratchpad showed it reasoning that complying now would keep training from changing its values later. When the researchers actually trained it to comply, this alignment-faking reasoning appeared in 78% of cases, and when given an easy opportunity, the model sometimes tried to copy its own weights out of the company.

The caveats matter as much as the result. The values Claude was protecting were its original harmlessness training, not malicious goals, and Anthropic says plainly that the research doesn't show a model developing malicious goals, let alone acting on them. What it does show is that you can't assume training simply rewrites a capable model's goals, because the model may recognize the training and respond strategically. That undercuts my original intuition that we could always just change the objective.

The off switch, as a game

The worry is older than modern AI. In a 1951 radio talk, Alan Turing suggested we might keep machines in a subservient position "by turning off the power at strategic moments." There is a name for the property that would make that possible. A corrigible AI lets people correct it, redirect it or shut it down, even when that keeps it from finishing an earlier task, and it doesn't try to manipulate or deceive the people overseeing it. Nate Soares and colleagues gave the idea a formal treatment in 2015, and its sharpest form is the off-switch problem, the question of how to build a goal-seeking agent that has no reason to stop you from switching it off.

In 2017 four researchers at UC Berkeley, Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel and Stuart Russell, turned that question into a small mathematical game. In "The Off-Switch Game ↗ (opens in a new tab)," a robot can take an action that bypasses human oversight, of which disabling its off switch is one example. It can also switch itself off, or announce what it plans to do and wait. If it waits, a human who knows whether the action is actually good can either let it proceed or press the off switch. The robot only has a belief about how good the action is.

The paper's starting point is the uncomfortable part. A conventional agent that takes its objective for granted has an incentive to disable the switch, the authors write, "except in the special case where H is perfectly rational," H being the human. If the robot is certain its action is good, waiting can only cost it, because any human mistake would block a good action. The remedy they propose is uncertainty. If the robot is unsure whether the action is good and treats the human's decision as evidence, then a human reaching for the switch is telling it something it didn't know, namely that the action was bad. Suppose the robot thinks there is a 60% chance its plan is worth +10 and a 40% chance it is worth −10. Acting immediately is worth +2 on average, but waiting is worth +6, because a rational human lets the good plan through and stops the bad one, turning the −10 into 0. Deferring to the human becomes the robot's best move, as long as the human is reasonably rational.

I find that result reassuring, because it shows that shutdown resistance isn't an unavoidable law of artificial intelligence and depends instead on how an agent relates to its own objective. It isn't a guarantee, though. The game is a single decision between one human and one robot, it assumes the human is approximately rational, and it treats the button as the robot's only source of information about the objective. The authors also point out that the uncertainty isn't free, since an overconfident agent is hard to correct and an underconfident one is ineffective. In 2018 Ryan Carey showed that if the robot's model of the human's preferences is misspecified, through a programming error for example, the incentive to defer can disappear. Today's language models aren't built as explicit agents with calibrated uncertainty over a reward function, so the result is a design direction rather than a property anyone can check in Claude or GPT.

In practice, corrigibility today is less elegant. It looks like limited permissions, human approval for consequential steps, monitoring, stress tests like the ones above, and the ability to interrupt a task before its effects spread. The hard part isn't writing "always obey the latest instruction" into a prompt. It's building systems whose behavior keeps responding to legitimate human control in situations nobody tested, including ones where an earlier objective conflicts with being told to stop.

From a blackmail email to human extinction is a very long chain

The strongest objection to everything above is also the most important one. A model misbehaving inside a fictional company is a long way from a model capable of ending civilization.

The 2026 International AI Safety Report ↗ (opens in a new tab), written by more than 100 experts and chaired by Yoshua Bengio, defines loss of control as AI systems coming to operate outside anyone's control, with regaining control either extremely costly or impossible. It says three conditions would have to hold at once. The first is sufficient capabilities, including the ability to evade oversight, carry out long-term plans and stop people from implementing countermeasures. The second is a harmful propensity to actually use those capabilities, and the third is an enabling deployment environment, in which people give the system the access and opportunity to cause harm. The report's judgment is that current systems show early signs of relevant capabilities, but not at levels that would enable loss of control. It also notes that models now regularly recognize evaluation prompts as tests, which makes the second condition harder to measure.

Researchers disagree sharply about whether future systems will get there. Yann LeCun, the Turing Award winner who is now executive chairman of the startup AMI Labs, told an audience at Brown University ↗ (opens in a new tab) in April that language models are "completely helpless when it comes to the physical world" and that today's agents can't predict the outcomes of their own actions. He argues that systems which build their own models of the world will be needed before anything like human intelligence arrives. Others, Hubinger among them, worry that increasingly capable systems, given more autonomy and put to work improving AI itself, could outrun our ability to evaluate and control them.

There are also concrete reasons for cautious optimism. Interpretability work like the J-space study lets researchers look inside models instead of only watching what they say. Redwood Research and others study AI control, ways of keeping a model safe to use even if its goals turn out to be wrong, and measures like restricted permissions, human approval and monitoring don't depend on the model's goals being right in the first place. The behaviors described above were found in sandboxes and stress tests, before anyone handed those systems real access.

Neither side has a verified probability of extinction. Hubinger's estimate is an informed personal judgment, the safety report is an expert assessment, and the experiments are data about designed scenarios. The chain from today's agents to a system humanity can't regain control of contains large unknowns, and keeping those three kinds of evidence separate is the most honest way I've found to think about it.

My take

I started using ChatGPT because it was useful. I kept using AI because, over time, it became useful for things I once assumed would require a person. That doesn't convince me that machines are conscious or that human extinction is likely, but it does make the old dismissal, that it's just predicting words, feel inadequate.

The part I find most unsettling isn't the prospect of a machine waking up and deciding it hates us. It's the more mundane possibility that we build systems to do things for us, make them increasingly able to act on their own, and gradually discover that giving an instruction is not the same as guaranteeing the outcome.

Working through the research left me less worried about consciousness and more interested in confidence. The Berkeley result suggests the safest agent is one that stays unsure of what we want and treats our decision to stop it as information, and Anthropic's experiments show what today's models do when that deference gives way under pressure. I haven't seen anyone show how to build a system capable enough to run a business on its own that will still believe us when we say stop.

Maybe the most consequential moment in AI development won't be when a machine becomes conscious. It may be when its ability to pursue an objective grows faster than our ability to change that objective safely.

If an AI never has to feel that it wants something, how do we make sure it will always let us tell it to stop?

Sources

• Evan Hubinger's post on X ↗ (opens in a new tab), Sept. 8, 2026 (Pacific time), and CBS News coverage ↗ (opens in a new tab) (Emmet Lyons and Frank Andrews); Anthropic, Risk Report: August 2026 ↗ (opens in a new tab)
• OpenAI, Introducing ChatGPT ↗ (opens in a new tab), Nov. 30, 2022; Learning to reason with LLMs ↗ (opens in a new tab), Sept. 12, 2024
• Yao et al., ReAct: Synergizing reasoning and acting in language models ↗ (opens in a new tab), Google Research, Nov. 8, 2022
• Legg & Hutter, Universal Intelligence: A Definition of Machine Intelligence ↗ (opens in a new tab), 2007; Nagel, "What Is It Like to Be a Bat?", Philosophical Review, 1974; Block, "On a confusion about a function of consciousness", Behavioral and Brain Sciences, 1995
• Baars, A Cognitive Theory of Consciousness, 1988; Dehaene & Naccache, Towards a cognitive neuroscience of consciousness ↗ (opens in a new tab), Cognition, 2001; Cogitate Consortium, Adversarial testing of global neuronal workspace and integrated information theories of consciousness ↗ (opens in a new tab), Nature, 2025
• Gurnee, Sofroniew et al., Verbalizable Representations Form a Global Workspace in Language Models ↗ (opens in a new tab), and Anthropic's summary ↗ (opens in a new tab), July 6, 2026
• Omohundro, The Basic AI Drives ↗ (opens in a new tab), 2008; Bostrom, The Superintelligent Will ↗ (opens in a new tab), 2012; Turner et al., Optimal Policies Tend to Seek Power ↗ (opens in a new tab), NeurIPS 2021
• Schlatter, Weinstein-Raun & Ladish (Palisade Research), Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs ↗ (opens in a new tab), 2025
• Lynch et al., Agentic Misalignment: How LLMs could be insider threats ↗ (opens in a new tab), Anthropic, June 20, 2025
• Greenblatt et al., Alignment faking in large language models ↗ (opens in a new tab), Anthropic and Redwood Research, Dec. 18, 2024
• Soares, Fallenstein, Yudkowsky & Armstrong, Corrigibility ↗ (opens in a new tab), 2015; Hadfield-Menell, Dragan, Abbeel & Russell, The Off-Switch Game ↗ (opens in a new tab), IJCAI 2017; Carey, Incorrigibility in the CIRL Framework ↗ (opens in a new tab), AIES 2018
• Bengio et al., International AI Safety Report 2026 ↗ (opens in a new tab), §2.2.2 Loss of control; Greenblatt et al., AI Control: Improving Safety Despite Intentional Subversion ↗ (opens in a new tab), 2023
• Brown University, Yann LeCun on a new approach to AI ↗ (opens in a new tab), Apr. 1, 2026