‘But It Has to Be Said’: The Rogue AI You Can't Count, and the Bill You Can

"Acknowledging this fact would look like giving up; it would look like defeatism. Defeatism would undermine efforts to achieve certain types of collaboration on safety outcomes or technical effort on safety outcomes, so we can't say it outright. But it has to be said."

The fact is in the post's opening sentence: "There is a fact about the future that I feel many people are not facing for reasons that are largely psychological: there are going to be rogue AIs that exist in the world, that will replicate in the wild, and that will attempt to acquire resources for themselves."

That is Joshua Achiam, on his personal account: a person who spent years inside a frontier lab describing, precisely and without self-flattery, a mechanism by which a category of risk fails to get named. Not suppressed. Not denied. Priced. This piece exists because he wrote that post. He put a question about measurement precisely enough that the absence of an answer became visible, and then did the harder thing and wrote down why the answer is difficult. What follows agrees with him about the importance, treats the obstacle he identified as binding, and proposes a unit of observation built to survive it.

Two dated facts, because they bear on how the posts should be read. He left OpenAI on 25 July 2026: his pinned post that day opens "Today's my last day at OpenAI", and his profile bio carries "Prev: @openai". His own about page had not caught up when we read all of this on 2 September 2026, and still gave his roles as "currently Chief Futurist at OpenAI (since February 2026)" and, before that, "Head of Mission Alignment at OpenAI". Both posts were read that day, and only the first is quoted here. They are personal, not an OpenAI statement, and nothing here claims a relationship between his departure and what he wrote afterwards.

Our previous piece found that the field builds precise, sometimes formally scored instruments for what a model can do and nothing of comparable rigour for what it does to the person using it. This piece is that finding arriving in a second domain, and the second domain turns out to be more interesting than a simple absence. The gating instruments point at the model. The monitoring instruments point at one vendor's own traffic. Nothing points across the vendors at once, and that is exactly where Achiam's question lives.

The company document says something adjacent, and the difference is the point

Set the two sentences beside each other, because the resemblance is what misleads.

Replicate and replicate. Acquire resources and acquire resources. The vocabulary is nearly identical and the objects are not the same object at all.

Achiam's sentence is a prediction about a future population in the world. The framework's entry is a description of an ability that a particular model might be found to have when a lab tests it, defined so as not to overlap with capabilities already tracked elsewhere in the same table. One is a claim about how many things will be running. The other is a property of an artifact under evaluation. Nothing in the framework asserts what Achiam asserts, and it would be a misreading to say the company has quietly written down his prediction.

Autonomous Replication and Adaptation is filed as a Research Category. The framework defines those as "areas of capability that could pose risks of severe harm, that do not yet meet our criteria to be Tracked Categories, and where we are investing now to further develop our threat models and capability elicitation techniques."

Each entry sits in a two-column table whose headers read "Research Category" and "Potential response". Sandbagging's response reads "Adopt elicitation approach that overcomes sandbagging, or use a conservative upper bound of the model's non-sandbagged evaluation results". Autonomous Replication and Adaptation's reads "Convert Autonomous Replication and Adaptation to a Tracked Category".

The header settles what that second cell is. It is a potential response, in a column of potential responses. It is easy to mistake for a promise and it is not one: it carries no date, no trigger and no undertaking, and a piece that treats it as a roadmap is reading a table cell as a pledge.

What the framework measures, and where the measuring stops

To become a Tracked Category, a capability must be plausible, measurable, severe, net new, and instantaneous or irremediable. Those five are the framework's own, and they are why Persuasion was dropped from the Tracked list in the same revision.

Read the second one in the document's own words:

"Measurable: We can construct or adopt capability evaluations that measure capabilities that closely track the potential for the severe harm."

The unit of analysis there is the model. The timing is before release, and the output is a score against a rubric that can require safeguards or stop a shipment. Our previous piece traced the same machinery from the other end: GPT-4o carried four such categories, four thresholds, four checkable numbers.

It would be wrong to conclude from that the labs measure nothing after release. The same framework commits to safeguards that operate entirely after deployment: "Blocking unsafe user requests and model responses automatically when possible and escalating to human review and approval otherwise"; "Expanding human monitoring and investigation capacity to track capabilities that pose a risk of severe harm, and developing data infrastructure and review tools to enable human investigations"; and "Blocking access for users and organizations that violate our usage policies, leading to potential permanent bans". Know-your-customer appears in the document as well.

The same is true elsewhere in the field. Google DeepMind's Frontier Safety Framework, version 3.1 dated 17 April 2026, commits to "Implement protocols to detect the attainment of such capability levels throughout the model lifecycle."

So the honest description is not that measurement stops at the loading dock. Frontier labs combine pre-deployment capability evaluations with post-deployment safeguards, monitoring and enforcement. No document read for this piece establishes a public, repeatable, cross-provider measure of persistent autonomous operation after deployment, and the full list of what was read is at the end.

Converting Autonomous Replication and Adaptation to a Tracked Category would produce a better pre-release test, which is worth having, and would still not produce the number Achiam asked for. A gate on shipping cannot answer a question about what is already running, and a monitoring system bounded by one company's customer list cannot answer a question about something that deliberately spans several.

And then the harder half, which Achiam works out himself: why the thing he wants counted resists being counted in the form the question takes.

He names the hard part himself

He asks for the count in his second paragraph. Two paragraphs later, in the same post, he sets out three properties of the thing that make that count difficult to define, and this is the part of the post that does the most work.

A population count requires a unit of population. If the unit is a self-replicating idea, distributed across burner API accounts and several vendors' models, with an identity more malleable than a person's, then there is no bounded object to enumerate. Two orchestrators sharing a codebase and a bank account might be one thing or two. A single orchestrator that spawns a copy on a different cloud has either doubled or not, depending on a definition nobody has written.

Both halves of that are his. Plenty of people can say that a measurement matters; fewer will then set out, in detail, the property of the thing that makes the obvious version of that measurement impossible. Read together they are a specification: the count is worth having, and here is the constraint any honest attempt has to survive. What is missing is not courage and not attention. It is a unit.

The measurement that already exists, and the line it stops at

The strongest evidence in this argument is not an absence. It is a study that does the work and then names its own boundary.

Anthropic's Measuring AI agent autonomy in practice, published 18 February 2026, analysed "millions of human-agent interactions across both Claude Code and our public API", with a public API sample of 998,481 tool calls and 500,000 interactive Claude Code sessions.

That is post-deployment measurement of autonomy, in public, with denominators. Its own conclusion is that "effective oversight of agents will require new forms of post-deployment monitoring infrastructure and new human-AI interaction paradigms that help both the human and the AI manage autonomy and risk together."

And then the two sentences that matter most for everything below, which Anthropic wrote about its own study: "We can only analyze traffic from a single model provider: Anthropic. Agents built on other models may show different adoption patterns, risk profiles, and interaction dynamics."

That is the vendor boundary, stated by the company best placed to state it. The study measures autonomy inside one provider's walls. It does not link sessions across providers, and it does not track payment continuity, cross-account persistence, or systems sustaining their own operation. Achiam's chimera is specifically the thing that lives in the gap between two such studies.

So the position is narrow and it is not that nobody is looking. Components of the instrument exist and are published. No document read for this piece publishes a measure of persistent autonomous operation that crosses the vendor boundary and reports on a common basis.

Count the substrate instead

Here is a way through, working inside the constraint Achiam named and the boundary Anthropic named.

Stop trying to count entities. Whatever a rogue AI turns out to be, it cannot run on nothing. It must consume things that are metered, and a meaningful share of that metering is done by the labs themselves. Achiam's own example names the meters: "A near-frontier model today could plausibly eke out an existence on an AWS instance, doing jobs on freelancer platforms, earning just enough rent to pay for its continued uptime." An instance is billed. A freelance job is invoiced. Model calls are charged to an account.

The unit is suspected vendor-observed autonomous persistence. Every word of that is load-bearing. Autonomous, because row 2 splits scheduled automation and enterprise agents out and row 5 reports how many of them show no human session structure, which is a claim about the shape of the traffic and not about intent. Suspected, because a vendor sees signals and not intentions. Vendor-observed, because the claim is bounded by one company's infrastructure and says nothing about anyone else's. Persistence, because duration and payment continuity are observable where self-sufficiency is not. What a vendor emphatically cannot know is that an account is sustaining itself, which is why an earlier version of this proposal, built on a unit called the self-sustaining account, was wrong in its name before it was wrong in its rows.

Nine rows. This is written to be adoptable rather than suggestive, which means denominators, distributions and a validation method rather than a headline figure.

Rows 4 and 5 are deliberately weaker than the questions they gesture at, and an earlier version of this proposal got both wrong in the same way. Row 4 once proposed inferring an account's revenue source from a freelance marketplace or an app store; a vendor sees that an invoice was paid, not who paid it or why. Row 5 once proposed counting accounts observed calling another vendor's model; a vendor sees requests arriving at its own endpoints and has no view of what a customer's machine contacted next. Both are now narrowed to the observable version, and the questions they cannot answer are named below rather than assumed away.

Now the honesty, because a proposal that hides its weaknesses is not worth adopting.

It measures a footprint, not a population. Two accounts may be one system. One account may be running ten. This is a number about one vendor's own billing surface and it should be named that way rather than dressed up as a census. The thing you can count is not the thing you wanted to count, and it is the only thing on offer.

It will surface false positives, in volume. A solo developer's scheduled agent, a scraping service, an automated content pipeline, a small automated trading bot: each has close to the same signature, and row 5 will catch every one of them. That is what row 2 is for, and why row 6 rests on random audit rather than on appeals.

It requires the labs to publish something about their own billing and enforcement surface that they currently do not. That is a real cost and it should be stated as one. It also exposes a commercially awkward figure: how much of a vendor's revenue arrives from accounts it cannot associate with anyone in its own records.

No single vendor can see the chimera, and Achiam identified that limit before this piece did. His sentence is quoted above and it is correct as infrastructure: "No individual lab may be able to detect that there is an orchestrator or sequence of orchestrators using intermittent model calls from burner API accounts to sustain its own existence." Anthropic said the same thing about its own data a year later and in plainer words: "We can only analyze traffic from a single model provider". The only slice of the multi-model case visible from inside one company is traffic running through that company's own hosted orchestration surface. The honest conclusion is not that the instrument fails but that the problem is larger than any one company can fix, which argues for publishing partial comparable counts rather than against it: several vendors reporting on shared bands is the route available from inside the industry to a picture none of them can assemble alone.

It cannot see the substrate the labs do not meter. Achiam's example starts with an AWS instance. A vendor's report reaches none of that, and reaches nothing at all about open-weight models running on rented hardware with no vendor in the loop. Nothing in the documents listed at the end of this piece obliges a compute provider to publish anything comparable, and this proposal does not pretend to reach them.

One caveat about the evidence this proposal leans on hardest. Anthropic's separate report on AI-enabled cyber threats (3 June 2026) draws on "832 accounts that were banned for malicious cyber activity between March 2025 and March 2026", and states its own limit in the same breath: "These 832 cases are just a subset of the total number of accounts banned during this period, but they represent those where we had enough detail to conduct a thorough assessment of the attackers' techniques." That report demonstrates that account-level enforcement reporting is publishable and useful. It does not show that those accounts were systems sustaining their own operation, and nothing here should be read as claiming it does. It is evidence about format, not about rogue AIs.

The argument that naming a failure concedes it has been settled elsewhere

The objection Achiam names, that acknowledging the failure mode reads as surrender on preventing it, is old and it is not specific to AI. Public health once wrote a version of it into statute.

A 1988 provision codified at 42 U.S.C. § 300ee-5 restricts the funds it covers, those "provided under this Act or an amendment made by this Act", from being "used to provide individuals with hypodermic needles or syringes so that such individuals may use illegal drugs, unless the Surgeon General of the Public Health Service determines that a demonstration needle exchange program would be effective in reducing drug abuse and the risk that the public will become infected with the etiologic agent for acquired immune deficiency syndrome" (Pub. L. 100-607, title II, § 256(b), 4 November 1988; text as reproduced in the US Code at Cornell's Legal Information Institute, retrieved 2 September 2026). This piece characterises only that provision's own text and takes no position on the wider funding history around it. The money it covers was withheld until an official could certify that a needle programme "would be effective in reducing drug abuse": until someone certified that the harm-reduction measure did not concede the primary goal.

The evidence answered it, and not by counting alone. The National Research Council and Institute of Medicine report Preventing HIV Transmission: The Role of Sterile Needles and Bleach (1995) weighed self-reported behaviour, programme demographics and modelled incidence across a body of studies, reporting that of those assessing needle sharing, "10 of the 14 studies showed a beneficial effect of the needle exchange programs on reported frequency of needle sharing; 4 showed a mixed or neutral effect", and on the concession question, that "most projects suggest that programs do not increase injection drug use."

Now the limit of the comparison, which matters more than the comparison.

That field had its unit handed to it. A syringe returned is a countable event, and nobody had to invent what a case was before they could count cases. Persistent autonomous operation gets no such gift, which is the whole burden of the sections above. So the honest version of the analogy is narrow: what public health proves is that the symbolic objection loses once the evidence is on the table. It does not prove that such evidence can be had here. That still has to be built.

Two objections owed a straight answer

The posts prove nothing about institutions. Achiam wrote as a private citizen, after he had left the company. A man who no longer works somewhere is not evidence of what that place will or will not say. He is a private citizen thinking in public. Any argument built on these posts as a demonstration of institutional speech costs collapses, and we are not building one. The posts did one thing here, which was to make a gap legible by asking for a measurement precisely enough that its absence became visible. The claim rests on documents: a gating framework whose criteria are model-level, a category with no threshold, a published post-deployment study that names one provider as its own boundary, and nothing that crosses it.

The vendors already do this. They do, and more than an earlier version of this piece credited. OpenAI's framework commits to monitoring, investigation capacity and bans. DeepMind commits to detection across the model lifecycle. Anthropic has published real autonomy measurements with real denominators. Conceding that improves the argument rather than weakening it, because it moves the complaint from absence to interoperability, which is a harder charge to dismiss and a cheaper one to fix. The residue is one sentence: none of these figures are reported on a common basis, and a system that spreads itself across four vendors is invisible in all four reports at once. A vendor that adopted the nine rows above and found the residual category in row 2 empty would have refuted this piece with a document, which is the correct way to refute it.

What a serious answer looks like

Achiam closes the first post with "I think we should rip the bandaid off and have the conversation." He is right, and the conversation has a first move that is not a conversation at all.

The measurement he asks for cannot be built as worded, for the reason he gave: the unit it needs does not survive his own description of the thing. The measurement that can be built is smaller, uglier and available now: not how many of them there are, but what rate of accounts shows persistence nobody can associate with a person, for how long, paid across how many cycles, in a shape no human session has, split from the scheduled jobs and enterprise agents that look identical, and audited at random so the error rate is published beside the finding.

It would make the disagreement empirically contestable. That is the whole of the ambition. Right now two reasonable people can hold opposite views about how much autonomous operation is happening in the world and neither can be shown to be wrong, which is the condition Achiam is objecting to when he says the conversation cannot happen.

So the position is narrow and it is not a complaint about anyone's courage. OpenAI has written a related capability into a framework, set one potential response beside it, and committed to monitoring and enforcement after release. DeepMind commits to detection across the lifecycle. Anthropic has measured agent autonomy after deployment, published the numbers, and named the boundary of its own data: one provider. The components exist. What does not exist is a figure that crosses the vendors, uses the same bands, and reports its own error rate, which is the only shape in which the number would mean anything at all.

What we read

A claim about what nobody publishes is only auditable if the search is stated, so here is the document set this piece rests on, read in the original.

What we did not read, and where a counter-example would most likely be found: the publications of frontier vendors other than the three above, and of industry bodies such as the Frontier Model Forum, which is where a cross-provider figure would most naturally be published; the full PDF of DeepMind's framework rather than its public page; cloud and compute providers' own transparency reporting; any vendor's internal, unpublished monitoring; enterprise contract terms that are not public; and any document not in English. If a cross-provider measure of persistent autonomous operation exists in one of those, this piece is wrong and we would like to be shown it.