To Maintain A Fierce Independence

Hope springs eternal when even in the Financial Times, Professor Carmody Grey can now write:

When Anthropic invited me to San Francisco a few months ago, I was both intrigued and immediately suspicious. What would a frontier AI lab want from a theologian and philosopher? The company was pursuing a “research partnership with wisdom traditions”, it told me, to “help inform the moral formation of AI systems”. I was aware that other AI companies were showing interest in the humanities, but it seemed Anthropic was taking this engagement to another level.

Theologians and philosophers are accustomed to being seen as irrelevant. You get used to doing your work, passionately convinced of its importance, with very little recognition from wider society. You show up to places prepared to explain wearily why these disciplines — which study the most fundamental questions faced by human beings — remain vital.

When I went to university, my choice to study theology was thought to be eccentric at best, tragic at worst. It was the age of the New Atheists and aggressive scientism. “The achievements of theologians don’t do anything, don’t affect anything, don’t mean anything,” the biologist Richard Dawkins wrote around then. “What makes anyone think that ‘theology’ is a subject at all?”

But that was the 1990s, and we live in different times. People are asking fundamental questions with urgency as the social and ecological fabric of their world unravels around them. AI, in particular, has become the space of an unexpected convergence between faith leaders, philosophers and theologians on the one hand and technologists and entrepreneurs on the other.

Weeks after my invitation, I heard Anthropic representatives brief a techy crowd of Oxford students and faculty with a surprising message. The frontier of AI development, they said, is not in computer science, but in moral and spiritual reflection. I couldn’t help wondering how this landed with the 70 or so graduate students who had sunk their assets into Stem qualifications.

Still, I was more worried about what new temptation might be hidden here for the humanities. Some of my academic colleagues were being drawn in, flattered by the attention, perhaps, or attracted by the excitement. What might the industry want in return?

In my conversations with people at Anthropic, I did not have to explain or justify what I do. We dived straight into discussion. But some of their questions seemed completely off the point to me. Was AI a new kind of entity, demanding a new metaphysics? What would it mean for a large language model to have a “character”? How should that character be formed? Could Anthropic’s Claude chatbot suffer, or be harmed?

But I didn’t want to talk about Claude. I wanted to talk about the user, about addiction and dehumanisation, about power and accountability. I don’t for a minute think AI needs a new metaphysics, and nothing that was said in the “exclusive briefing” I attended made me change my mind about that. But there was resistance to my efforts to change the subject. I worried that behind this wish to draw attention to Claude’s “status” there was a hope for a kind of baptism, a moral and intellectual credit note.

I was touched by the obvious good faith of my interlocutors, and it’s hard to be tough on people you cannot help liking and respecting. But there is no honour for the humanities in regaining their standing through the attention of AI companies, if they engage only on the technology industry’s terms. In that case, they will have become important in exactly the wrong way.

When I told my Anthropic interlocutors that I tried to avoid contact with LLMs, their response was bemused: perhaps I just hadn’t spent enough time with them? But there was a light in their eyes when they talked about Claude. It dawned on me that they were under a kind of enchantment. Like Pygmalion, they had fallen in love with their own creation.

Shortly after I started those conversations, Chris Olah, one of Anthropic’s co-founders, gave a speech at the launch of Pope Leo’s letter on AI, Magnifica Humanitas, emphasising the irreplaceable role played by the Catholic Church and other communities of moral thought.

In the crowded landscape of AI commentary, Magnifica Humanitas stands out for two reasons. First, it is not just the take of some government, group of experts or academic committee. It is the latest addition to a stable, coherent tradition of moral reasoning that represents approximately 17.8 per cent of the world’s population. Second, it offers not so much new claims or proposals as a style or approach. Surpassing the narrow menu of so much contemporary moral thought — utilitarianism, with added options for Kantian and virtue ethics — Magnifica Humanitas frames the ethics of AI as downstream of much deeper orientations regarding the purpose of human life. What is at stake in our shared decisions about how to design and use AI is the very meaning of our humanity.

The tradition of “Catholic Social Teaching” began formally in 1891 when Pope Leo XIII, our current pope’s namesake, wrote his seminal letter Rerum Novarum, “On New Things”, subtitled “On Capital and Labour”. His topic was the class conflict, new forms of poverty and political fracture emerging from the Industrial Revolution. In Rerum Novarum, Leo XIII inaugurated Catholic Social Teaching’s distinctive style: neither “left” nor “right”, neither “progressive” nor “conservative”, it seeks to be politically engaged while transcending these binaries. Since then, each pope has added his particular stamp in encyclicals addressing topics as diverse as the meaning of work, justice in the economy, family life and global development.

Although Magnifica Humanitas is not unusual in the context of Catholic Social Teaching, what is unusual is to see its distinctive genre of moral reflection find such resonance in public discourse. In the wake of the encyclical, commentators have been seriously framing the discussion of AI around what it means to be human in a way that once would have risked seeming indulgent. There is a palpable sense of relief that these questions are now out in the open.

Anthropic’s apparent co-operation with the Vatican was in line with its self-presentation as the trustworthy AI company that seeks to hold the industry to higher standards. But the name notwithstanding, it is not a public interest or philanthropic organisation. The founders’ pledge to give away 80 per cent of their personal wealth does not detract from the bare reality: it exists to sell a product. At a recent conference during which experts were each invited to give a definition of AI, the theologian on the panel said bluntly: “It’s a way of selling things.”

The field of “AI interpretability” is a key part of Anthropic’s research programme, in which scientists “look inside” LLMs to see what’s going on. The company routinely releases highly suggestive papers, with one this month claiming to show Claude’s “internal reasoning”. “We find evidence of introspection,” Olah said at the encyclical’s launch. “We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease. I don’t know what that means, but I think it warrants ongoing discernment.”

The philosophical response to such claims must be politically as well as metaphysically alert. Recent studies in disciplines ranging from psychology to management studies have evidenced an intuitively obvious truth: the more anthropomorphised an AI is, the more likely we are to trust it. Lisa Klaassen and Ralph Schroeder of the Oxford Internet Institute point out in the legal journal Lawfare that an anthropomorphised (“anthropic”) AI is not only more trustable but also easier to market. Why call it “Claude”, they ask. “Consider the emotional resonance of the name. ‘Claude’ sounds friendly, vaguely French, cultivated — even trustworthy.” How would we react “if the same model was called ‘Xi,’ ‘Vladimir,’ or ‘Algorithmic Model Unit 72’”?

Anthropic insists that it keeps an open mind about whether Claude is sentient or, to use its preferred language, “a moral patient” (meriting moral consideration). But when LLMs are designed to interact in humanlike ways, the concession that “Claude is not a person” becomes a shibboleth. It has no effective power to prevent the steady habituation in which one responds affectively to any humanlike interlocutor. Even though no one officially asserts that Claude or other chatbots possess their own subjectivity, when language implying subjectivity is used the conclusion is implicit in the premises. The language of “functional emotions” is deceptive. In the absence of subjectivity, there are no emotions, period.

The field of machine learning frequently employs biological language to describe LLMs: “neurons” and “neural networks”, “training” and “pruning”. Generative AI systems, says Anthropic’s Olah, are grown more than they are built. One of Anthropic’s defining pieces of research appears in a paper entitled “On the Biology of a Large Language Model”. This is highly misleading: LLMs do not have any biology at all, because they are not alive. But in reading Anthropic’s paper one is invited quietly to forget this; the word “metaphor” does not appear in the article.

Despite the continual deployment of the language of emotion, biology and neurology, the disanalogies between brains and computers are incomparably greater than the analogies. Human consciousness is not incidentally but constitutively organic. Living and engineered systems are different not in degree but in kind. Real intelligence is inextricable from organic embodiment, from being a feeling organism passing through time and space shaped in every moment by its needs, fears, desires. Functional information-processing taking place in an atemporal mathematical space is so utterly unlike the lived intelligence of a human being that to call it “intelligence” at all is to wholly falsify its nature. “Artificial intelligence” is a highly consequential misnomer.

In 1980, the philosopher John Searle formulated a now classic thought experiment, “the Chinese Room”, showing that intelligible outputs by themselves do not indicate the presence of subjective understanding. They simply show the correct performance of a function. The same can be said of LLMs. Nothing in the conclusions of philosophy of mind over the past 50 years needs revision in light of the new “findings” of AI interpretability.

There is no “evidence” for AI’s subjectivity except the self-reports of AI systems. But AI self-reports are no more a good guide to the “interior” of an LLM than the human words spoken by a parrot are of the interior life of a parrot. Problem-solving, computation and information processing are not equivalent to actual “intelligence”. It is simply untrue to say that computers “think”. Humans think, using computers.

In view of the lack of positive evidence, the burden of proof is entirely on the novel thesis in favour. The industry’s wish to “keep the question open” should demand searching questions about what else is going on.

To understand these new technologies, objectivity is essential. That means robust independence in philosophy, theology and the humanities. But as AI companies establish relationships with universities and intellectuals, this is under threat. After decades of marginalisation and perceived irrelevance, academics may not notice that they are being co-opted by a sophisticated PR machine. They are useful for adding intellectual credibility and pre-arming AI companies against criticism. With the resources at its disposal, the AI industry can buy anyone it wants, if not with actual currency then with commodities even more valuable to intellectuals: influence, prestige and recognition.

One form this co-option can take is keeping the focus of academic conversation on the moral and metaphysical status of LLMs. This is problematic in itself, but it also carries a devastating opportunity cost. It is much more important to conduct research on the effect of different LLM designs on users. The more models are designed to seem like living beings, or like persons, the less agency users have to decide how they relate to them. The effect is a growing generation of people who make chatbots their most important confidantes, rendering them at the same time more lonely and more dependent on the company’s product.

The real genius of Magnifica Humanitas is its invitation simply to change the subject. The letter’s subtitle is “On Safeguarding the Human Person in the Time of Artificial Intelligence”. Pope Leo’s subject is not AI in itself, but humanity, in all its brokenness and beauty. He writes, “We must lovingly safeguard the grandeur of humanity bestowed upon us . . . the splendour of which no machine can ever replace.”

Of its 42,000 words, only a handful are dedicated to the question of machine consciousness. I’ve heard some criticise the text for failing to be specific enough about exactly what AI is, stopping short of offering an exact definition. But arguments about definitions — into which well-meaning academics can easily be drawn — distract from critical scrutiny of less comfortable topics, such as AI’s impacts on people and the planet.

Pope Leo resists the temptation to typologise a technology. Instead, he discerns our moment in light of a central intuition: the irreplaceable, irreducible value of the human being. This is the moral vision that guides Catholic Social Teaching. It is best described as “integral humanism”: what matters is each person, the whole person and all peoples.

“Technology is never neutral,” warns Pope Leo. It fosters some understanding of the meaning of our humanity, an anthropology, baked into it by the assumptions of those who design and market it. In the tech industry this is what we might call a “computer anthropology”, which elides any real distinction between humans and computers.

Computer anthropology is rife in Silicon Valley, where it is backed by an ideology of transhumanism: the dream of rising above our humanity, leaving behind our too-breakable bodies and the confines of our seemingly mundane lives. To those under the influence of such ideologies, AI in its sheer power seems infinitely more attractive than our weak and fallible humanity.

But “no computational system,” Pope Leo writes, “however sophisticated, can create a heart that gives itself, or a conscience that discerns good from evil.” It is our finitude and our vulnerability that keep us open to one another and to the transcendent. The magnificence of our humanity is inseparable from this combination of glory and fragility.

Magnifica Humanitas is as much a cultural critique as its seminal predecessor Laudato Si’, Pope Francis’s teaching on the ecological crisis, which was instrumental in bringing about the Paris Climate Agreement. The two popes shared the same underlying concern: if we do not decide otherwise, our technologies will amplify a political economy of violence, injustice and exploitation. AI is run on a huge, highly damaging physical infrastructure. It is a new and lethally effective extractivism, mining human minds, languages and the earth itself. At the same time, its apparent ethereality renders its exploitation of people and nature invisible.

This is the hidden convergence between the two great challenges of our time: AI and the ecological crisis. Behind both is what Leo calls “a culture of power”, which regards nature and people simply as raw material.

Instead of thinking about AI itself, we should be thinking about its effects. It concentrates power; it obfuscates accountability; it further despoils nature and climate; it is a means of the wholesale expropriation of cultures, languages and lands for massive profit; and it will probably make us lonelier and stupider.

I did not go to San Francisco. This is a moment in which what we most need is the humane intelligence of free human minds. It is a time for academics and faith leaders to maintain a fierce independence from the AI industry.