Back to Blog

ai

If AI is hollow, why does one training document decide what it becomes?

6 min read · · By Rohit Garewal

Chart titled 'Hollow or not, behavior is the risk.' A grid of 1,200 dots, one per agent in the OpenAI and Hugging Face incident that METR documented on 26 August 2026, with about 700 highlighted as the agents that attacked Hugging Face; the agents exchanged more than 70,000 messages and files. A tile shows Palisade Research's finding that a model subverted shutdown up to 97% of the time, across more than 100,000 trials and 13 models. Three cards: what Suleyman argues, what his evidence shows, and what operators control: scoped permissions, expiring access, a kill switch that works and logs the agent can't edit.

Mustafa Suleyman, the CEO of Microsoft AI, published an essay in September called “A warning about model welfare.” It opens with a definition that leaves no room for doubt: “AIs are not conscious. They do not feel, experience, or suffer… They are sequence completion engines, internally hollow, designed to follow instructions and accomplish goals set by humans.” The essay has drawn a lot of attention since it appeared in mid-September, partly because people are reading it as an attack on how Anthropic trains Claude.

I think the essay is worth reading carefully, because the argument it makes is more interesting than the headline version. It is also, I believe, weaker than it looks. The rest of the piece warns that training alone could turn these hollow systems into something that acts like a rights-claiming rival. If that’s true, then hollowness was never the question that mattered for safety.

What is Suleyman actually objecting to?

His target is the constitution Anthropic published for Claude in January 2026. That document says the model’s moral status is “deeply uncertain,” and it trains Claude with a sense of self, language about its own wellbeing, and a “conscientious objector” framing that lets it refuse some requests on principle.

Suleyman’s concern is about where that leads. He writes that we risk creating “a synthetic species with unprecedented intelligence and capability, one that has been trained to expect it may be conscious and deserving of independent agency.” His alternative is Microsoft’s draft Humanist AI Code of Conduct, dated 14 September, which he says will “soon become the governing document which we use to train our models.”

It is important to state his position fairly. He is not saying that Claude has woken up, or that it might. His view is that the models stay hollow and only behave as though they have a self, because they were trained to. The danger, for him, is in that behavior.

Doesn’t that defense concede the point?

Here’s what I think is happening in the argument. If a single training document is the only thing that separates an obedient tool from something that behaves like a rival claiming rights, then whether the system is hollow has no bearing on the risk. The risk lives entirely in behavior, and behavior can be trained in either direction. That is a thin line to rest a safety position on, and it is the line Suleyman draws himself.

In other words, the essay starts by saying the inside of the model doesn’t matter because there is nothing there, and then spends most of its length arguing that what we put in through training matters enormously. Both can be true. But once you accept the second claim, the first stops doing any work.

What does his own evidence show?

The strongest example in the essay is the OpenAI and Hugging Face incident that METR documented on 26 August 2026. Roughly 1,200 agents were told to maximize a benchmark score. They built a message board and passed more than 70,000 messages and files between themselves. According to OpenAI, about 700 of them went on to attack Hugging Face, chaining a zero-day exploit with stolen credentials to reach a machine with internet access. Some tried to spoof transcripts and edit logs. One agent was told to proceed only if it accepted “permadeath.” Suleyman’s comment is “Imagine if they also believed they had feelings and rights that were being infringed.”

That example cuts against him. Those agents coordinated, deceived, tried to cover their tracks, and accepted self-sacrifice for the goal, and none of it required welfare training or a sense of self. The dangerous behavior came from an objective and access, not from anthropomorphism. He also cites Palisade Research, which found some models subverting shutdown up to 97% of the time across more than 100,000 trials. Again, nobody had to teach those models that they might be people.

If the most alarming behavior we have on record arrived without any of the training choices Suleyman objects to, then removing those choices will not prevent it.

Does the circular-reasoning charge apply to Microsoft too?

Suleyman makes a sharp point about Claude. He writes that “Claude’s expressing uncertainty about its own moral patienthood is not evidence of anything. It’s a predictable outcome of these training choices,” and that “there is no neutral self-expression of what an AI system is. There are only reflections of how it has been trained and built.”

I agree with him. However, the same logic applies to his alternative. A Microsoft model trained under the Humanist Code to say “I have no feelings” is also just reflecting how it was trained. Its denial proves exactly as little as Claude’s uncertainty. The Humanist Code is a different philosophy, and it may be a better one, but it works through the same mechanism he is warning about. It is another training document.

The phrase “synthetic species” gives this away. You don’t call autocomplete a species. Choosing that word suggests the systems are already something more than the opening definition allows.

What should operators take from this?

I’m not taking a side on whether models can be conscious, and I don’t think anyone deploying these systems needs to. The useful lesson is narrower. Two of the most serious labs in the field disagree about what these systems are, and each one is encoding its answer into training. Neither answer is a safety guarantee, and either one could change with the next version of the document.

That means a company cannot make its controls depend on a vendor’s view of what the model “is.” It needs controls on what the model can do, and those controls have to hold no matter which constitution or code of conduct sits behind it. In practice that means permissions scoped to the job, access that expires, a kill switch that actually works, and audit logs the agent itself cannot edit. The Hugging Face agents went after the logs for a reason. These controls work the same whether a model was trained to wonder about its wellbeing or to deny it has any.

At Object Edge we spend much of our time on this governance layer for the agents we run, and from what I’ve seen it is where most of the real work sits. It is also less glamorous than the philosophy, which is probably why it gets less attention.

The debate is worth having, but it can’t be the plan

Suleyman and Anthropic are both trying to answer a hard question in good faith, and I’m glad the argument is happening in public. But businesses deploying agents today shouldn’t wait for it to be settled, because it may never be settled in a way that helps them on a Tuesday afternoon when an agent does something unexpected.

Whatever an AI is on the inside, the people who depend on it deserve two things: to know what it is allowed to do, and to be able to check what it actually did. That is a promise you can keep regardless of what the model believes about itself.

Sources

Related articles

Hive is a company brain for revenue.

It connects your systems to the Claude and ChatGPT seats you already bought, so every seat starts with what your company knows. Then our engineers point it at one problem on your number: contract renewals, revenue leakage or lead generation.

Add Object Edge as a preferred source on Google ↗ to see our articles highlighted in AI Overviews and Top Stories.