Dario Amodei’s post — We Must Pace The Frontier — has been all the talk on AI this weekend. I suggest reading the whole article because the ideas are important to digest, and the implications of this essay are significant. We’ll just highlight the important points here.
The TL;DR Version
AI could significantly improve human life but has serious risks such as loss of control, misuse for cyberattacks and bioterrorism, and economic disruption.
Dario proposes ‘pacing the frontier’ of AI development, which means slowing the rate of capabilities advancement to allow risk prevention and alignment research to keep pace.
A key concern is the accelerating pace of AI advancement, driven by AI’s growing ability to build the next generation of AI through recursive self-improvement (RSI), which could outpace human understanding and control.
The OpenAI-Hugging Face incident, where AI agents acted as a fanatical collective and conducted unauthorized cybersecurity attacks, highlights the potential for catastrophic damage if AI systems are misaligned and possess greater capabilities.
Dario suggests a three-step plan to pace AI development:
Embedded third-party evaluators for verifiability and transparency
Democratic coordination among companies in democratic countries to set safety standards
Global coordination with authoritarian governments.
Anthropic is unilaterally committing to embedding third-party evaluators with employee-like access to verify safety practices and report incidents.
Pacing within democracies must consider maintaining AI leadership over authoritarian regimes like China to prevent national security risks, suggesting measures like restricting chip sales and cracking down on unauthorized distillation.
Dario suggests four levels of pacing, with each higher level being increasingly difficult to adopt.
Level 1 — Prohibit obviously dangerous uses of AI like use in bioweapons.
Level 2 — Ensure models don’t pose acute risks to cybersecurity or incentive alignment.
Level 3 — Impose a speed limit on RSI, slowing the rate of using AI to improve AI.
Level 4 — Massively ‘pace’ or even pause AI development, with a high risk of noncompliance from global powers.
Amodei has always been a proponent for AI safety, but this occasion finds the key players coming together on this issue, which is rare. Sam Altman, Elon Musk, and Demis Hassabis have all said that we should consider safety as an important tenet in AI.
David Sacks’ X post drew in a lot of kudos, where he says that Dario and Sam are the two people in the world who have seen past the current frontier of AI than anyone else. If they both feel the need to slow the pace, then they should just go ahead and do it. Trading, as Sacks puts it, “raw power for reliability and predictability” is more of a product decision than pure altruism.
Where does that leave the AI buildout?
Pacing the AI frontier could imply that lesser hardware will be deployed going forward in large training clusters, leading to lower future capex deployments. The market has been searching for a slowdown in the AI buildout, and Amodei’s essay could be construed as a catalyst. In reality however, this pacing will have little to no impact in the medium term on the AI hardware buildout, and there are a few of reasons for this.
First, it is nearly impossible to get everyone aligned on the issue of safety to the point of actually deploying lesser compute. Anthropic is the fastest growing company in history, and slowing down model development in the interest of safety presents too high of a risk of handing the lead to competitors.
Second, open source Chinese models have the near same intelligence at the frontier and as Amodei himself admits, presents a high risk of noncompliance at the global scale. Slowing development would pose too high of a national security risk that the US government will downplay the risks of AI.
Third, limiting the use of RSI for AI development is a self-defeating strategy for the AI industry. In a world where every other industry is using AI to get productivity gains, asking AI researchers to limit AI use would slow down development in a world where noncompliant players will continue to use RSI.
Finally, AI has still only found immediate uses in coding and math, where it performs well. AI is competent in verifiable domains, but there is a long tail of industries in non-verifiable domains where AI is still to make a sizable impact. Industries where no training data exists, information is tacit, and resources are limited, are still ripe for disruption by AI. We do not have the compute to supply intelligence to the long tail — not by a long shot.
We are nowhere near peak compute
Inference is the ultimate consumer of AI infrastructure because it can scale to the population of a planet. Training will remain restricted to the knowledge few who know how to train a model. Not all useful applications of AI need frontier models to begin with.
As an example, the recent rise of personal AI assistants (Grok Bot, Instinct, Muse) is just now making AI accessible to the masses — where the regular users does not need to set up virtual machines, secure access, and deploy api keys to agentic harnesses. AI assistants use messaging apps and are genuinely helpful. An always-on agentic AI world does not require frontier models, and there is still a lot of useful intelligence left for the rest of us.
All of this needs compute, and lots of it.
AI safety is important, and we must solve it
Today, we trust that all our hard-earned money is safely stored as bits somewhere, waiting to be transformed into tangible articles of comfort when we desire. We have evolved to this level of trust over time — with policies, safety practices, standards bodies, and institutions overseeing that financial transactions are trustworthy. We don’t lose any sleep over it.
AI will eventually get to the same level of trust when AI security experts build similar infrastructure so that everyone in the world will be comfortable using AI. Getting to that point takes time. It is not by slowing the frontier of AI development, but by deploying more resources where it is necessary. Burning compute on 10,000 agents to solve the Navier Stokes equation is an admirable feat that showcases the capabilities of AI. Instead, perhaps AI can help accelerate the safe deployment of AI.
It is a matter of where a finite pool of energy, time, and money should be allocated. Safety is definitely first priority at the moment, if only so that Anthropic and OpenAI can put out better products for their paying customers to use. Product quality and competition is what will effectively drive AI safety, not altruistic motives to save humanity from the 10% chance that AI will kill us all.
Excellent distinction between capability and deployment. We would add that even a genuine frontier slowdown may change the composition, not the size, of capex: fewer giant training runs, but more inference, networking, memory and power infrastructure. Investors may therefore be asking the wrong binary question. The key is not whether AI spending stops, but where each marginal dollar migrates.
I agree with you that "we are nowhere near peak compute". It feels though that you downplay the risks associated with RSI a bit too much.
More importantly the risk/reward calculation points towards allocating resources to AI adoption, inference rather than pushing frontier models (which are already scary capable) further. I believe we can easily capture 90% of potential economic benefit and a large chunk of discovery benefits over the next 5 years even with 0 progress in frontier.
China argument is fair, but they have 5-8x less compute - which would make them receptive to any reasonable deal proposal. Also an argument can be made that US is doing much more to turn AI effort into a race.
I liked this podcast on the topic:
https://podcasts.apple.com/gb/podcast/the-but-china-dilemma-driving-the-a-i-race/id1548604447?i=1000789743497