Machines of Loving Grace and the marginal returns to intelligence

The setup

Dario Amodei has published a long essay on what he thinks a good outcome from AI would look like. He explains at the start why he has avoided writing about benefits before: the people who discuss radical AI futures tend to do so in a sci-fi tone that makes the claims easier to dismiss, and he wants to avoid grandiosity. He then defines the system he is writing about. Powerful AI, in his usage, is smarter than a Nobel Prize winner across biology, programming, mathematics and engineering, has the interfaces a remote human worker has, can work autonomously for hours or days, has no body but can direct robots and lab equipment, and can be run as millions of copies at ten to a hundred times human speed. He summarises this as a country of geniuses in a datacenter.

The headline claim follows from that definition. He proposes that AI-enabled biology and medicine could compress the progress human biologists would have made over the next fifty to a hundred years into five to ten, and calls this the compressed 21st century. Four other areas get the same treatment: neuroscience and mental health, economic development, peace and governance, and work and meaning. Most of the commentary we have seen this week is about the ambition of that list. We think the more durable contribution is the part that pushes back on it.

The marginal returns framework

Amodei argues that in the AI age we should be talking about the marginal returns to intelligence, that is, how much an extra unit of intelligence buys you in a given domain once the other inputs are held fixed. He then lists the things that limit those returns. The speed of the outside world: experiments, cell cultures, clinical trials and hardware fabrication run at their own pace regardless of how clever the person waiting for them is. The need for data: some questions cannot be answered without measurements that do not yet exist. Intrinsic complexity: some systems are chaotic or unpredictable enough that no amount of computation resolves them. Constraints from humans: laws, ethics and institutions will limit what an aligned system is allowed to do. And physical laws, which set floors and ceilings that intelligence cannot negotiate with.

This is a good list, and it is a better tool than the compressed-century claim it is attached to. It gives you a procedure. For any proposed application of powerful AI, ask which of the five constraints binds first, and estimate the speedup from that rather than from the raw capability. In drug discovery, the binding constraint is the trial, and the essay's own argument is that intelligence attacks it by finding better measurement and intervention techniques rather than by thinking harder about the same data. In mental health, the constraint is data and complexity. In governance, it is human institutions, which is where the essay gets into trouble.

What we would add is that the list is also a list of research questions. Each constraint is a claim about where the returns to intelligence flatten, and each could be tested at small scale with the models we have now. Does a more capable model actually reduce the number of wet-lab iterations needed to reach a result, or does it just propose more candidates? That is measurable, and the answer would tell us more about the compressed century than any amount of argument.

The entente section

The section on peace and governance is where the essay stops being a forecast and becomes a proposal. Amodei's current guess at the best way to secure a good outcome is what he calls an entente strategy, in which a coalition of democracies seeks a clear advantage in AI, secures its supply chains, and uses that advantage as both stick and carrot: military superiority to deter, and access to AI's benefits offered to a wider coalition of nations that give up competing with democracies. The optimistic version, in his phrase, is an eternal 1991, a world in which democracies have the upper hand and Fukuyama's dreams are realised.

There are at least three things to argue with here, and the essay acknowledges some of them. The first is that the plan requires extraordinarily wise decisions about the balance between carrot and stick, which is an admission that the strategy's success depends on judgement the plan does not supply. The second is that a strategy of clear advantage is also a strategy of racing, and racing is the thing the safety community, Amodei's own community, has spent a decade warning about. The third is that the constraints-from-humans item on his own list applies here with full force. Institutions do not move at the speed of the models, and an entente built on temporary technical superiority is a bet that the superiority lasts longer than the institutions take to catch up.

Reading the two halves together

The essay is strongest when the marginal returns framework is applied and weakest when it is forgotten. The biology section applies it carefully and arrives at a claim that is large but argued. The governance section sets it aside and arrives at a claim that is large and asserted. We do not think that is an accident. The constraints on intelligence in biology are physical and can be reasoned about. The constraints in geopolitics are other people, and they do not hold still for the analysis.

That is also why the essay is worth reading rather than summarising. The definition of powerful AI, the five constraints, and the compressed-century target are a coherent framework that anyone can use to make their own estimates, and most of the estimates we would make with it are lower than Amodei's. The entente section is a policy position from the head of a frontier lab, and it should be read as one.

What we would want next

We would like someone to take the five constraints and turn them into a scorecard for a specific field, listing for each proposed AI-driven advance which constraint binds and what the realistic speedup is once that constraint is priced in. Biology is the obvious candidate because the essay has already done half the work. If the compressed century survives that exercise, it is a much stronger claim than it is today. If it does not, we will at least know which decade we are actually talking about.