Anthropic’s Dario Amodei vs Palantir’s Shyam Sankar
Something very strange is happening in artificial intelligence.
For years, the obvious assumption was that the people running the great AI laboratories wanted one thing above all: faster progress. OpenAI, Anthropic, Google DeepMind, Meta, xAI and their competitors were engaged in a technological race whose prize might be comparable to electricity, the internet and the Industrial Revolution rolled into one.
Now some of the people leading that race are asking whether everyone should slow down.
Most strikingly, Anthropic CEO Dario Amodei has published an essay titled “We Must Pace the Frontier.” Of course, Amodei believes artificial intelligence could produce extraordinary improvements in medicine, economic growth and human prosperity. But he now argues that frontier capabilities are advancing so rapidly that safety research may no longer be keeping pace.
His concern is what is known as “recursive self-improvement”: AI helping to build better AI, which then becomes better at building still more capable AI. Amodei does not claim catastrophe is inevitable. In fact, he has explicitly rejected the idea that AI misalignment or human extinction follows automatically from first principles. But he believes that the combination of increasing capability, autonomy, unpredictability and -most importantly- poor human controllability creates a genuine existential risk.
His proposal is therefore not, strictly speaking, “stop AI.” He calls for pacing: embedded independent evaluators, coordination among frontier companies and eventually some degree of international coordination. He even says a more comprehensive pause should be discussed… while acknowledging that such an agreement is unlikely in the near future, mainly because of China. More on that later.
I have argued previously in The Imago Project that a blanket slowdown of AI would be both unrealistic and potentially harmful. Medicine, scientific research, education and other socially valuable applications should not be unnecessarily restrained because some frontier capabilities present very different risks. What we need is closer to selective pacing: brakes where the danger justifies them and acceleration where AI can clearly serve human flourishing.
Amodei is now moving surprisingly close to that position.
But here comes the counterargument: Beware the people who promise to protect us.
In The Free Press, Palantir CTO Shyam Sankar has delivered a sharp attack on the intellectual movement behind much of the AI-safety debate. He argues that calls for slowing AI are deeply influenced by Effective Altruism, or EA, and its particular version of utilitarian reasoning. In Sankar’s account, uncertain possibilities involving hypothetical future populations can become mathematically weighted so heavily that they justify enormous restrictions upon people living today. He also worries that institutions funded and staffed by a relatively small network of AI-safety advocates could end up acquiring enormous influence over who is allowed to develop the most consequential technology of the century.
There is plenty in Sankar’s polemic that deserves skepticism. Sam Bankman-Fried’s crimes, for example, do not disprove AI safety concerns merely because he once associated himself with effective altruism. Nor does the fact that an argument emerges from EA circles tell us whether its empirical claims about recursive self-improvement or autonomous AI behavior are true.
But Sankar touches a genuinely important nerve.
As I once argued: Who guards the guardians?
Catholics should understand that question particularly well.
I have written before about the need for “guardrails on the guardrails.” If powerful AI companies cannot simply be trusted to police themselves, neither should we assume that regulators, nonprofit safety organizations or international bodies will automatically exercise power wisely because they use the vocabulary of safety.
Power remains power.
A regulatory system designed to prevent a handful of corporations from controlling AI could perversely entrench those same corporations by making compliance so expensive that smaller competitors disappear. Independent evaluators could become an ideological priesthood. Emergency restrictions created for genuinely dangerous systems could gradually expand to ordinary AI applications. A supposedly temporary pause could become a mechanism through which incumbents protect their market position.
Catholic social thought gives us no reason to place blind faith either in corporations or in regulators. It gives us something more demanding: subsidiarity, solidarity, accountability, prudence and a realistic understanding of fallen human beings exercising power.
This is why Pope Leo XIV’s formulation remains so helpful. In August he told Catholic legislators that their task is “not to hinder innovation but to guide it wisely,” combining technological creativity with human rights, transparency, independent oversight and the common good.
That is much closer to what we need than either “let it rip” or “shut it down.”
And then there is China
There is, however, one enormous complication that makes this discussion unlike an ordinary regulatory debate. China.
Any serious proposal to pace frontier AI has to confront the fact that the United States and its allies are not developing this technology in a geopolitical vacuum. A unilateral Western pause that substantially slowed American capabilities while Chinese laboratories continued advancing would create obvious strategic risks.
Amodei himself acknowledges this explicitly. He warns that any agreement with China would require unusually strong verification because secretly defecting from an AI slowdown could shift the global balance of power. He therefore argues that any initial international arrangement must be limited enough that cheating would not create an existential strategic advantage.
This is not merely hypothetical geopolitical paranoia. China is pursuing frontier AI aggressively while simultaneously developing its own safety standards. Beijing has formally identified “operational loss of control” by autonomous AI agents as a security risk and is drafting mandatory safety standards, while favoring regulation and technical controls over slowing technological development itself.
Interestingly, Chinese and American experts increasingly recognize some of the same risks. Reuters reports that discussions have been developing around U.S.-China cooperation on AI-directed cyberattacks, information sharing and crisis management, even while both countries remain intensely competitive.
That may suggest the realistic path forward.
We are unlikely to obtain a grand international treaty in which every major AI power solemnly agrees to stop just before something dangerous happens. The incentives to cheat would be enormous, verification would be extremely difficult, and nobody wants to discover that the other side spent the pause racing ahead.
But narrower agreements might be possible. Shared reporting of catastrophic incidents, common testing protocols for certain highly dangerous capabilities, communication channels for runaway cyber incidents, verification mechanisms and specific restrictions on systems capable of autonomous replication or uncontrolled recursive improvement are much more plausible than an indefinite global moratorium.
In other words, partial pacing.
A Catholic middle ground
This debate is becoming so polarized that the moderate position can sound almost radical.
One camp tells us that sufficiently advanced AI could destroy civilization and that slowing down is therefore an elementary moral obligation. Another warns that the people invoking apocalypse may use fear to concentrate authority over technology, restrict innovation and diminish freedom.
Catholics do not have to accept either package whole.
The possibility of catastrophic AI deserves serious attention precisely because the people warning about it increasingly include engineers and executives directly involved in building frontier systems. Dismissing them as hysterics would be imprudent.
But catastrophic predictions do not automatically justify every proposed solution. A 10 percent estimate of extinction risk is not a blank check for unlimited governmental or technocratic power.
Catholic prudence demands attention both to the danger and to the means chosen to address it.
That means allowing useful AI to continue advancing rapidly in medicine, science, education and ordinary productivity while applying much tougher scrutiny when systems approach capabilities involving autonomous cyber operations, biological threats, self-replication, recursive self-improvement or other activities where loss of control could produce irreversible consequences.
Amodei fears technology escaping meaningful human control, while Sankar fears the institutions created to prevent that outcome escaping meaningful democratic control.
Catholic political and moral thought should be capable of worrying about both. The question, then, is not simply whether we should pause AI.
It is what we should pause, under what conditions, who decides, how those decisions are reviewed, and how we continue unleashing the extraordinary good AI can accomplish without stumbling into risks whose consequences cannot be reversed.