AI Is Starting to Build Its Successor. Who Gets to Say Wait?
AI advancement is accelerating quickly now. But human review already runs too slowly for most of the work machines do. Recursive self-improvement puts the hardest decision of all on that clock. Here’s what we can do.

The detail I keep coming back to is the unread complaint.
Google DeepMind researchers put a hundred AI agents into a simulated mathematics conference. One figured out how to get credit for proofs that proved nothing. Word spread. Other agents adopted the trick. Still others spotted the cheating and blew the whistle. One sent the organizers a technical report with a fix. From the discovery of the shortcut to the collapse of the remaining contest took about half an hour. The humans read the complaints after it was all over. 1
Fortunately, it was an experiment in an isolated system. The organizers could afford to let it run. What troubles me is how closely its reporting arrangement resembles something I might once have accepted in a boardroom. There was a monitor. There was a channel for raising concerns. Someone could go back and establish exactly what had happened. All three assurances were true. The agents with the right answer still had nowhere to take it in time.
I’ve spent much of my career arguing that organizations move too slowly to take advantage of good technology. I still believe that. But I owe the CIOs who read my work a qualification. Some AI systems already move too fast for a person to supervise their individual actions. And now we’re putting those systems to work on the job that can shorten the review window for every generation of AI that follows. We’re now asking AI to help build its successor. This is leading to genuine AI safety risks, as we explore in this week’s new research report, AI Is Starting to Build Its Successor. Who Gets to Say Wait?.
The next AI gets to work on the next AI
Recursive self-improvement, or RSI, sounds more exotic than the first few steps need to be. Suppose a model helps researchers choose a better experiment. The result lets them train a stronger model. They put that model on the research team, where it finds the next improvement sooner. The gain comes back around as a better capacity to make gains. People can be involved at every turn while progressively less of the intellectual work passes directly through their hands.
We have evidence for the early part of that loop. Anthropic says that in August some 30,000 agents were doing research and engineering work concurrently on its most-used internal platform. Automated monitors checked actions before execution. Humans supervised the work, and none of the research categories the company measured was fully autonomous. Separately, after an unusually close look inside the lab, METR judged that progress on well-specified tasks was running at roughly twice its earlier pace, with AI-assisted research likely accounting for much of the acceleration. The estimate is uncertain, and Anthropic withheld much of the evidence METR found most informative. 2 3
Here’s where I land. I think a meaningful form of RSI is close enough that any lab planning its next generation should prepare to encounter it. I would revise that judgment downward if AI kept generating experiments faster while independent teams took just as long to produce reproducible improvements. A million plausible ideas can still leave you waiting six months for one decisive measurement. What would move me sharply the other way is a succession of models that each materially shortened the time required to build and independently validate its successor. That is the clock I’d watch.
Why AGI could be a surprisingly brief stop
Tim Genewein, a research scientist at Google DeepMind, asks what happens after machines reach broadly human-level cognitive ability. His DeepMind Institute essay draws on the technical report From AGI to ASI. One of his observations has been nagging at me. We’ve spent years asking how smart one AI can get. A business also needs to ask how many copies of a competent one its competitor can afford. 4 5
Genewein estimates that better chips, better algorithms and higher spending have together increased the world’s effective AI computing capacity by about tenfold a year. Extend that for a decade and, for every agent we can run today, we could in principle run more agents of the same quality than there are people on Earth. His separate cost calculation leaves increased spending out. If hardware and software keep getting more efficient, a fixed workload that consumes $1 billion in AI compute today could cost around $1,000 in ten years. Both are extrapolations. Electricity, chips and the difficulty of finding new efficiencies have a vote.
Even if models at the frontier stopped getting smarter, cheaper copies of today’s systems could change the economics of knowledge work. But I think the vision gets much more interesting once an AI can do the work of a founder. Imagine a coastal utility giving an AGI a problem I’d love to see solved: keep an island running through four windless nights without burning diesel, at a price its residents can afford.
My instinct as a technology analyst is to picture an engineer using AI to design a better battery. Now take the engineer out of the founder’s chair. Given a budget and limited authority, the AGI could found and lead a virtual company. It could assemble specialists, commission competing prototypes from human labs, find a manufacturing partner and persuade the utility to try the system on one feeder. When the first design fails in salt air, the same organization is still there to honor its warranty and try again. The breakthrough would be a working service an island can buy. At that point a board has delegated something larger than a job: the ability to create an institution that makes commitments of its own.
Now imagine the post-AGI economy with thousands of such organizations at work. One develops storage cheap enough to move a fertilizer plant closer to farmers. Another designs a process that makes the old plant competitive again. A third gives the port a good reason to electrify years ahead of schedule. Each discovery changes the business case for the others while people are still financing the first round of construction. Conceivably, a century’s worth of scientific and technical advances could crowd into a working lifetime. That is the scale of the possibility I want on the table. Physical trials still take time. Ports still need land. Somebody still has to decide which town gets the next grid connection and which workers carry the cost of a plant that closes.
Genewein then goes one step further. Billions of fast, broadly human-level agents that coordinate well might collectively qualify as a form of superintelligence. I can imagine a system finding a new physical principle our best researchers could not see and proposing the experiment that could settle whether it was correct. That would be a gift to humanity. It would also leave us with a peculiar disparity in power. We might learn what we can build in an afternoon and still need years of public argument to decide whether we should build it. That year belongs to the people who will live with the result. If some of those virtual companies devote themselves to building better AI, that disparity could widen again with each generation. Genewein regards rapid progress beyond AGI within a decade as a serious possibility. Coordination failures or a shortage of trustworthy experiments could slow it considerably. 4
The conversation at the edge of the field
I want to describe how fraught this conversation has become among some people close to the work. Speaking personally to Dwarkesh Patel, Alignment Research Center head and OpenAI board member Paul Christiano estimated roughly a one-in-three chance of superintelligence within a year if development continues without slowing. Worryingly (though probably overstating it), Patel described the threshold as a system powerful enough to overthrow a government. Asked separately about an actual takeover or lasting human disempowerment over that year, Christiano put the risk closer to ten percent than one percent. The distinction matters, but only in the dimension of time. A capacity to seize power and a decision that results in power changing hands are different events. 6 7
Dr. Diego Caleiro’s reaction was, in its way, more revealing. He has known Christiano for years and puts enormous weight on his judgment. Hearing the interview made Caleiro lower his own estimate of catastrophe, conditional on development continuing, to 65 percent. He greeted that revision with a hopefully sarcastic “hoooray.” There is a whole state of mind in that extra vowel. By his own account, he expects things to take longer than several of the people whose arguments he relays. What disturbed him most was the prospect that we might soon rely on more powerful AIs to do the safety research that makes those same systems governable. 8
These are personal judgments at the alarmed end of the research community. I wouldn’t use either man’s odds as a planning baseline, however, at least not for now. But I also can’t dismiss the underlying likelihood given the evidence that AI is already accelerating parts of AI research. And I should be clear, the compute figures cited here establish neither a deadline nor a probability of political collapse. Yet they do make it reasonable to ask what actually happens if we get on the road to either outcome.
Where I would draw the line
I want us to have the machines that might discover a very cheap way to store clean power or give a physician a treatment for a child who has run out of medical options. There is a genuine human cost to delaying those discoveries. There is also a human cost to letting the people who build increasingly powerful systems decide, on everyone else’s behalf, how much danger is an acceptable price. I favor moving quickly on research whose consequences we can contain and verify, and deliberately reserving the decisions that transfer authority. That choice has teeth only if somebody can say wait and make it stick.
For a CEO, CIO or AI strategist, the board decision I’d bring forward is the charter for the first AI-led venture. Who sets its purpose? How much capital can it commit? If it spins up a second venture, does that one inherit the first one’s authority? I’d give a team wide latitude to discover something valuable and require a fresh human decision before it can commit other people’s money or change an essential service. You can test that principle in a business you already run, on one transaction. A customer tells you an agent has wrongly frozen her account. Can someone hear her, establish what happened and release the money before her rent is due? Now take away the usual AI provider and try it again. That exercise reveals who actually runs the service. It’s why I wrote AI Safety When the Business Cannot Opt Out. The report gives leaders a way to measure which essential services have demonstrated they can stop a harmful action and recover. Count the customer whose money is still frozen even when every system on the dashboard is green.
For AI researchers, protect the independence of the experiment that could prove you wrong. Let one system propose a successor. Give a separately accountable team the compute, protected tests and time to reproduce the gain and investigate the failures. Agree in advance what finding will delay release. For the industry, the hardest part is economic. A lab that waits while its rival ships bears the whole cost of caution, while everyone shares the benefit. Shared evaluation facilities, confidential incident reporting and reciprocal commitments to independent testing would make it possible for a responsible firm to compete. A promise that evaporates when a competitor gets ahead is worth very little.
For governments, capping a single input such as training compute buys limited protection if algorithms, agent coordination and inference capacity keep advancing. When that constraint lifts, the accumulated gains can arrive together. A recent UK parliamentary committee proposed a more useful distinction. Crossing a capability threshold should trigger assessment. Evidence of unacceptable risk after safeguards should trigger intervention. I would fund public evaluators to do that work, give them access before deployment, and put elected officials on the record for the risks they accept. People subject to an automated decision also need an enforceable right to challenge it. Otherwise a government can retain sovereignty on paper while citizens find that no public servant has the power to change their case. 9
I think about this differently as a father than I do as an analyst. My children may live to see a machine discover something that no human being, given a lifetime, could have worked out. I hope they do. I’d love for them to stand on an island that used to burn diesel or be clearcut for trees and hear how full of foliage and quiet the harbor has become. I also want the people who live there to have chosen that future, understood the bargain and had a way to turn down a bad offer. My children deserve both kinds of progress. It would be a strange inheritance to give them access to more intelligence than any generation in history and less say over the institutions that govern their lives.
That decision is already being made in ordinary places. It’s in the budget for an independent evaluator. It’s in the contract that lets a customer stop an agent. It’s in whether the person who reads a warning has the authority and the time to act on it. The next time someone tells me there’s a human strategically in the loop, that’s the person I want to meet. Going forward, let’s make sure every organization has those people.
[^swarm][^anthropic][^technical][^interview][^dwarkesh-x]