Nobody is ready for AGI: Miles Brundage leaves OpenAI

What he said

Miles Brundage, who ran policy research and then the AGI Readiness team at OpenAI, published a note on October 23 explaining why that Friday was his last day. The sentence that travelled is that neither OpenAI nor any other frontier lab is ready for AGI, and the world is not ready either. He adds immediately that this view is not controversial among OpenAI's leadership, and that it is a different question from whether the company is on track to be ready by the time it matters. It is worth reading the whole post before quoting that line, because the framing around it is careful and the line on its own is not.

He gives three reasons for leaving. Publishing constraints inside the company had become too much for him, though he says some constraints in industry are reasonable. He wants to be less biased, and says people are right to question policy ideas coming from a company with a financial stake in the answer. And he feels he has done most of what he set out to do. The team itself is being dispersed. The economic research group moves under the new chief economist, Ronnie Chatterji, and the rest of the projects go to Josh Achiam's Mission Alignment team.

The case for independence, as he makes it

The part of the post we keep coming back to is the second reason. Brundage does not say OpenAI stopped him from doing good work. He says that policy research produced inside a lab is discounted by the people it is meant to persuade, and that the discount is justified. That is a claim about credibility rather than about censorship, and it is a harder one to fix from the inside, because no amount of good faith by the researcher removes the conflict of interest from the reader's view.

His list of what he wants to work on next is long and unglamorous. Assessing and forecasting AI progress, regulation of frontier safety and security, the economics of AI, speeding up beneficial applications, compute governance and what he calls AI grand strategy. He plans to start a nonprofit or join one. The list reads like a description of the work a lab most needs done and is least able to do credibly.

What we learned making the same choice

This foundation exists because a few of us reached a version of that conclusion earlier and with lower stakes. We wanted to run evaluations and interpretability work whose results would be believed regardless of whether they flattered a particular model, and we concluded that the only reliable way to get there was to have no model to flatter. It has worked in the way Brundage predicts. When we publish a negative result about a system, nobody asks what we were selling.

It has also cost what he seems to expect it will cost. Independence means smaller compute budgets, no early access unless a lab chooses to grant it, and less information about what is actually happening inside the frontier. Some of the most important questions, such as how a lab's own internal evaluations behave in the months before a release, are visible only from inside. An independent group can study the artefacts a lab chooses to publish and can be honest about them, and that is a narrower thing than being in the room.

What independence does not solve

Brundage is right that readiness is a property of the whole world as well as of a company, and we think the honest corollary is that an independent nonprofit does not make the world ready either. It produces analysis that can be trusted. Whether anyone acts on it depends on governments, on the labs themselves, and on whether the analysis is good enough to be worth acting on. The credibility that comes from leaving is a precondition for influence. It does not supply the influence itself.

There is also a selection problem that nobody outside the labs can fix. The people who leave to do independent work are, by construction, the people who found the constraints intolerable. The people who stay are the ones who can live with them. If the field's independent policy research comes mainly from the first group, it will carry that group's priors, and the discount that Brundage worries about for industry research will have a mirror image for the research done by its former employees. The fix is boring. Independent groups need to be funded on terms that do not depend on which lab they left.

What we hope he does

The item on his list that we would put first is forecasting and assessment of AI progress. It is the piece where a lab's incentives are strongest, the piece where public claims are least often checked, and the piece where a well-resourced independent group could have a measurable effect within a year. If the new organisation publishes its forecasts with dates and scores them in public, it will have done something no lab has managed, and it will have made the case for independence better than any resignation note can.