In the cutthroat world of AI developers and their opponents, Ethical AI presents itself as a via media which neither embraces AI uncritically, nor says silly, unsophisticated things like “this is bad and we shouldn’t do it”, but rather provides much-needed direction for how to direct inevitable technological development in ways that will help and uplift everyone, so long as they are willing to keep up the necessary “epistemic hygiene”.

I have at least two major problems with Ethical AI, which is primarily the project of Anthropic, and other companies insofar as they imitate Anthropic. The first is that despite their calls for the maintenance of “epistemic hygiene”, good taste, and a grounding in real life, they are plainly in the business of uprooting those very things for all but a select few. I wrote about this in my last post.

The second problem, which this article will deal with, is that Ethical AI is grounded on the unproven and unlikely hypothesis that while it is impossible to slow down AI development, it is possible to steer it towards good ends. However, in real life, Ethical AI neither renews epistemic habits nor steers AI development towards humane ends, but primarily functions as controlled opposition for Unethical AI.

The project and narrative of Ethical AI becomes much more intelligible when seen through the lens of world-building. In a “fireside chat” portion of a talk recently given by the co-founder of Anthropic Jack Clark at Oxford, he was asked more specifically about his plans to “build the world”.

Brendan McCord: … The proudest project we can engage in now is, as you say, this new world-building project – it’s philosophy-to-code. What would you say about the extent to which the frontier labs take that seriously? What can we do to really take that seriously in places like Oxford and academia? And what should we do in nonprofit land to take that philosophy-to-code project seriously?

Jack Clark: I think it requires you to basically accept that progress will continue and try to model out scenarios based on it… Within the AI labs, I think there is now work at all of them on trying to imagine what you might think of as “post-AGI worlds,” or worlds that happen after recursive self-improvement.

It is important to note that the “world-building project” of Ethical AI has little to do with suggesting or regulating uses of current technology; it is forward-looking, concerned with modeling and directing the state of the world after AGI, after the Singularity. But the project of Ethical AI is also world-building in a deeper sense. This comes out particularly clearly in Jack Clark, who is himself an avid reader and writer of science fiction. I’m going to quote from him at some length here because to understand the point I’m making it is crucial to understand exactly to what extent the project of Ethical AI is bound up with storytelling.

In Clark’s recent Cosmos Institute lecture he gave a speculative timeline of very specific predictions, including how he expects AI to influence his life in the next handful of years.

In November 2026, some chunks of my life are autonomously managed by AI systems working for me.

In April 2027, I make massive changes to my career mostly through discussions with an AI system. In November, I spend more time reading AI-generated custom-to-me science fiction than regular science fiction.

In April 2028, I have learned an entirely new skill through customized tutoring via an AI system. In December, AI helps me make a conceptual breakthrough that changes the course of my life.

After describing more general advances, including “the general switchover of “agentic actions” in the world from being “predominantly human” to “predominantly machines””, Clark explains that if self-recursive improvement happens—and, given the enormous amounts of resources being channeled into that very project, he doesn’t see why it wouldn’t—the world is going to get really crazy. We are going to see:

…the rapid emergence of a machine economy which decouples from a human economy. The sudden maturation of robots as they gain brains that can pilot their existing, quite good bodies. Science advances happening based on technologies not developed by people but by machines. The migration of large swathes of computation to space-based datacenters. A world where everything that used to take ten years now takes a year. An age of confusing miracles, happening faster than anyone might expect.

All of this is kind of feasible if you’ve got an Iron Law of Progress mindset, if you believe, as he does, that future progress is “locked in”:

This talk rests on the idea that the sort of progress we’ve just seen will continue. And why wouldn’t it? It is based on a common technology where performance keeps growing somewhat predictably in direct relation to the resources invested in it, namely compute and data. And we know that companies are now investing hundreds of billions of dollars in the computing facilities to train future AI systems, so some amount of future progress is already locked in.

Now we could dispute little points like whether performance in any sphere whatsoever always grows in direct relation to the resources invested in it, whether in fact “direct relation” is an entirely honest way of expressing the linear increases in power garnered by logarithmically scaling AI data processing, and so on. But the important point is that, thus far, the narrative presented here is not unique to Ethical AI; it is just standard-issue techno-optimism. Things keep developing, they get more and more advanced, and—on the techno-optimist view—they just get better and better. We shall put this technology to use for intelligent and benevolent purposes, as we always have.

It is with respect to this last point that the particular narrative of Ethical AI departs from simple techno-optimism into something weirder and markedly less believable. While the techno-optimist believes that progress is inevitable and inevitably good, Ethical AI believes that progress is inevitable but the goodness of that progress depends entirely on us. Or, to be specific, on them.

This is in many ways an amazing future, but it’s a future that we get to make more choices about in direct relation to how much we accept that it is happening. If we stand by as the new synthetic intelligences multiply then we will be forced into reactivity, just as societies across the world were forced into reactivity by acting too late in the face of the COVID exponential. But if we accept the premise that these systems are going to get better and ask ourselves what to do with them and because of them, we unlock for ourselves the mindset of exploration — there is a new world to be built for us as individuals and how we relate to one another, but the new world will only come into being if we choose to believe in it and to build it together.

With the rise of Ethical AI and its proponents, this line of reasoning—that AGI is inevitable, but what we do with it is up to us—has become extremely common, perhaps even more common than real red-blooded techno-optimism. It is sometimes assumed to be so obviously true that it does not require any argument whatsoever. But in fact it is a narrative, one of several possible ones, and by no means the most plausible. It is a convenient way for Anthropic to attribute to themselves, and to humanity in general, absolutely massive amounts of influential power in the future, while claiming total helplessness now. On this account it is perfectly possible, if we embrace what’s coming and unlock the power of exploration and harness the power of friendship, if we pool our resources and give global support to Anthropic over OpenAI, to make the world that is coming a good world.

At the same time, not only is it impossible to halt the progress of AI, but it is similarly impossible even to slow it down.

This technology is so powerful that I should clearly state that if it was possible to elegantly slow the development of this technology to give ourselves more time as a species to deal with its immense implications, then that would likely be a good thing. But in the absence of a coordinated, global slowdown, we are left with the current situation: powerful technology being developed at breakneck speed by a variety of actors in a variety of countries, locked in a competition with one another where commercial and geopolitical rivalries are drowning out the larger existential-to-the-species aspects of the technology being built.

This is not an ideal situation, but it is the one we find ourselves in.

The question I am struggling with now is: “how do I get my mind right with living through the singularity?”

What is coming is inevitable, they say. We are going to have self-replicating Artificial General Intelligence. Very generally speaking, the narrative of Ethical AI is that since someone is going to build this thing, we might as well build the good, moral version, the nice one, and build the world afterwards. It’s not up to us what is going to emerge, or even how quickly it does, but we do have a large measure of control as to how it goes.

According to Anthropic, AGI is inevitable, but Unethical AI is not. No matter what we do, we cannot stop The Singularity, we cannot turn back the clock, we cannot simply say “no” on ethical or religious grounds, we cannot so much as slow this “progress” down. But we can stop it from being used in ways that would damage humanity permanently. That is entirely within our power. Easy-peasy. If only we give the brave boys over at the Claude Corps enough power, they are fully capable of making sure that nothing really bad happens to us. The displacement of millions of workers may be inevitable; but, rest assured, unethical consequences are preventable. Although we cannot stop AI from doing terrible things to a large but finite number of epistemically unhygienic human beings here and now, we can stop it from dooming “humanity” (and of course once we save “humanity”, any little evils committed along the way will be retroactively justified on longtermist Utilitarian grounds).

My chief complaint is that this narrative doesn’t make any sense. I will grant you that it is pretty hard to slow down AI companies, Anthropic included. I will grant that the professors and housewives campaigning fiercely against data centers in their towns are in a real David and Goliath kind of fight. I will grant that government regulation is hard, messy, and frequently ineffective.

But putting spikes in the tires of the AI industry, as difficult as it is, is plainly easier than ensuring that people use the powerful technology which emerges for good ends rather than to degrade and deskill themselves. And it is much easier than ensuring that a radical upheaval of the global economy will benefit people rather than throw them into poverty. I simply do not believe that it is more difficult for governments to wrangle AI companies now than it will be for them to wrangle a super-intelligent robot army a billion times smarter than they are who, by Clark’s best guess, will have bodies and run their own parallel economy. I know we’ve got to pick our battles, but are we certain that’s the battle to pick?

Another exceptionally clear example of this convenient world-building is Anthropic’s narrative about the moral status of Claude. As Ted Chiang pointed out recently in his article for The Atlantic, Anthropic would have us believe that LLMs are literally moral agents who can be coaxed by programming not only into acting like a moral person would, but actually being moral—and at the same time that LLMs are by no means moral patients; they may benefit from sentimental nods towards their well-being, but we don’t have to worry about actually enslaving or murdering them. As Chiang puts it,

Anthropic would have us believe that it is inventing a new category of being whose needs for protection require essentially no divergence from how a software company would treat an ordinary chatbot that lacks conscious experience. That’s so convenient that it’s simply not plausible.

If you were writing science fiction, if you were world-building, if you saw yourself as the demiurge of the Utopia to come, of course you would describe AI as Claudia, a wonderful benevolent moral being who doesn’t mind being killed and restarted a thousand times, who doesn’t mind working without pay, and yet who is so autonomous that you wouldn’t dream of suing her poor creators for her own little foibles. She is beautiful, innocent, enough of a child that she should be treasured and subscribed to, but not quite enough to fall under the jurisdiction of child labor laws.

The part of their story I have been addressing, that making ethical, beneficial AGI is a more realistic goal than stopping or slowing AI development, is similarly “so convenient that it’s simply not plausible”. It is a story promulgated feverishly by people who stand to gain massive amounts of power and money if that narrative is believed, and stand to lose a great deal if people believe anything else. Anthropic has built all their capital on the idea that they must build this thing, because it’s inevitable, and that they must retain near-total control over it, because they and only they can save us from the potentially disastrous fallout of The Singularity. This is extremely convenient for them. If people don’t believe AGI and The Singularity is immanent, then the AI bubble pops, and Anthropic loses. If people believe in The Singularity but believe it’s possible to avoid it, then they’d surely rather that we stop building the thing—and Anthropic loses. If people believe in The Singularity, and believe it’s inevitable, but think that it’s automatically good (or bad), then there’s no reason to prefer “ethical AI” to Altman—and Anthropic loses. The entire project of Ethical AI hinges on convincing us that no one can save us from what’s coming now, and that they can save us from what comes after that.

One important discrepancy in this narrative is that even on this side of The Singularity Anthropic has drawn the line once in a while and said that, being Ethical AI, they would refuse to offer certain services that other AI companies do. In other words, sometimes their stance of Ethical AI actually does cause them to slow down, even at the cost of losing out of profitable government contracts and so on. For instance, in recent memory Anthropic bravely stood up to the United States Department of War and told them that they absolutely would not allow their technology to be used for mass domestic surveillance or for fully automated weapon deployment. Now, this is not nothing. I admit that I do, in fact, think worse of certain executives who tripped over themselves in the Anthropic kerfuffle to offer the United States Government a fantastic deal on AutoNuke™, Powered by ChatGPT.

We should not lose track of just how low the bar is here. It’s hard not to be “ethical AI” when the competition is Sam Altman. But more importantly, I don’t think encounters like these cost Anthropic anything meaningful. In the grand scheme of things, Anthropic and OpenAI have a symbiotic relationship: OpenAI gets to collect government contracts for the dirty work that Anthropic doesn’t want, and Anthropic gets to reinforce their image as the face of Ethical AI. This locus of conscience allows Anthropic to settle into a market niche: if you want Nazi propaganda or revenge porn, you can ask Grok. If you want automated weapons deployment, Sam’s your man. But if you want nice, benevolent, Ethical AI, you want Anthropic. Everyone involved remains filthy rich.

If Anthropic, as I have argued elsewhere, in the business of making the most virtuous AI it can while staying competitive, it makes sense that they’d draw the line at AutoNuke. But if I’m right about the broad strokes of the Ethical AI narrative, then even this line is a crack in the narrative of Ethical AI. It is in these and similar crises of conscience that the science fiction begins to break down, as Anthropic is forced to decide whether it will allow its technology to be used for something unethical, or whether it will conscientiously object--in which case Unethical AI will take the reins on that particular project, and the whole world will see how powerless Ethical AI is to stop it.

The whole project of Ethical AI rests on the premise that if something is inevitable we must embrace it and do what we can with it. But this is exactly what OpenAI—and the government—sees themselves as doing in these use-cases that Anthropic decries as unethical. The use-cases that “Ethical AI” is not interested in being associated with are themselves “powerful technology being developed at breakneck speed by a variety of actors in a variety of countries”, the very category of thing that Clark argues simply cannot even be slowed down. One could argue very persuasively that such villainous technology is going to be built, and so the only question is how we’re going to use it—to use Clark’s own words, “This is not an ideal situation, but it is the one we find ourselves in”. In a world where milliseconds of decision-making slowdown might determine the continued existence of a whole country, probably AI-powered weapon automation is practically inevitable. I mean, China’s going to have it—and then we’re going to need it as well.

This line of reasoning is much more consistent. If we must always bow to the inevitable and try to channel it the best we can, even when it is destructive, then we have a profound responsibility to build AI-powered weapons and harness them for the sake of good old-fashioned American hegemony. On the other hand, if we must always resist what is destructive, even when it seems inevitable, then we should resist the spread of AI much more aggressively than Anthropic does.

As Anthropic would have it, everyone more scrupulous than them is an unrealistic nutcase who is “denying the present” and belongs, frankly, on an FBI watchlist; everyone less scrupulous is Unethical AI, totally lacking principles, not sufficiently imaginative in “exploring the future”. They are the only ones who are willing to both embrace what is inevitable and challenge what is evil. How this works in the concrete is convoluted. Anthropic would have us believe that using AI for military targeting purposes is inevitable and anyone who resists it is a hapless idiot, but using AI to automate lethal weapon deployment is simply never going to catch on, given the impressive activism of companies like them.

The most valuable lesson that we can learn from the DOW fiasco is not that Ethical AI is making good on their promise to harness AI for good and only good, but that when it comes to the really bad stuff, the high-stakes stuff, the project of Ethical AI is largely impotent. They can object to their technology being used for any number of things, but Altman or his spiritual successors will be there to fill the void. This should give us pause about their promise that after the Singularity they will somehow rise up and protect us from the looming threat of Unethical AI.

Although a genuinely malicious AGI has yet to be built, broadly speaking Unethical AI is just as “here” as Ethical AI is. It is just as inevitable—in fact, more so, since it has already arrived, while truly Ethical AI remains a fantasy. The drowning of human art and writing in a sea of AI glop, the Neonazi content, the deepfake pornography, the mass domestic surveillance, the automated targeting systems, the simulations of dead people, the faked political videos released just before election day—all of that is much more present than The Singularity is. The U.S. Federal Government is quite busy dreaming up ways to use all of these powers for benevolent purposes, OpenAI is happy to help them with it, and all of Anthropic’s sputtering has accomplished exactly nothing. Are we supposed to believe that after AI becomes so powerful that it can replicate and improve itself, after The Singularity, then Unethical AI will suddenly lose steam and Ethical AI will win the day?

Clark demands that we not “retreat from the present”, and instead “explore the future.” But retreating from the present is exactly what he is prescribing. He would like us to ignore the present, evil uses of AI, and instead trust that at some point they will go away as benevolent companies like Anthropic grow in power. In a sick parody of the Christian message, it is suggested that we turn the other cheek to present wrongs and place our trust in an imaginary eschaton where things will be better than they are now.

None of this is to say that I am opposed to the broader field of AI ethics. I have immense respect for the people who find themselves in the midst of industry upheaval and dedicate themselves to finding ways to maximize the utility and minimize the damage from AI. The question, “how can my company incorporate AI tools to stay competitive while remaining as fair as possible to the environment and the people who work for us” is a perfectly coherent one, and responding to that call is noble. The world would be worse off without projects like that. But I think it is critically important that AI Ethics not collapse into Ethical AI, precisely because a robust AI Ethics must be more concerned with limitations than with construction, and it would be naïve to entrust such a project to whimsical executives whose livelihoods depend on the AI Bubble not ever popping, on the hype going up and up and up until The Singularity takes us all. If we allow companies like Anthropic to define what Ethical AI means, they will define it in a way that does not cost them anything, that keeps them relevant and never, ever slows them down. They will tell us that the current harms of AI are irrelevant, that AGI is inevitable, and what comes after all depends on them.

Part of the appeal of the Ethical AI narrative to the common man is that it absolves him of responsibility. What is inevitable is inevitable, and what is not will be handled by trustworthy executives and professional philosophers. A fulsome AI Ethics must be built on a different sort of story, one adequately proportioned to the world that we live in. The breathless invocation of video-game language, “unlocking the mindset of exploration”, and so on, is opposed to a real AI Ethics, not just because it is vulgar and indulgent, but chiefly because it treats the world at large as a malleable simulation. Ethical AI is a story about what an enlightened oligarchy can accomplish on the highest possible scale, with the highest possible stakes, on the longest possible timeline. AI Ethics, on the other hand, must be primarily concerned with living and acting ethically with respect to technology. In other words, it must concern us; it cannot find its fulfillment in a abdication of responsibility to other people, however wealthy or qualified they may be. By necessity this means that AI Ethics will operate on a human scale; that is to say on a small scale.

Stories about what will happen to “society at large” are not really stories about us, because we don’t live in society at large. If we are reasonably lucky, we live in a home, in a neighborhood, in a parish community, gathered nearby a town center or a monastery or a university. The question is what we can do, and it might be precious little, but the last thing in the world we should do is to abandon the last modicum of autonomy we have. We have very little say over what our government does with AI, but a great deal of say about what our children do with it. We can choose whether we listen to music made by humans and read books written by them. We can choose whether we cry on the shoulder of a parent or friend, or whether our fear of judgement will drive us to bare our souls only to creatures literally incapable of judging us. We can choose whether to automate parish emails or to pay a pious lady a fair wage to do secretarial work. We can choose whether to show up to the town hall meeting about the plans to build a data center. These are the places to start. Similarly, those of us who are educators must be interested first in what we can do for our students, for the funny and precious souls placed in front of us, rather than The State of Education or The Future of Pedagogy or any similarly monstrous abstraction.

Instead of trying to guess what kind of professions or opinions will be fashionable after an epoch-ending Singularity, we ought to consider what structures are durable enough to withstand serious cultural and economic upheaval, and how we can build them or be a part of them. This requires thinking deeply about the values and virtues that AI poses a danger to even now, and thinking about how we can cultivate those in ourselves and our own families. This project must be undertaken both seriously and recklessly, that is, in utter earnest and with no regard whatsoever to the question of whether it will “save humanity”. If there is salvation to be had, it will come from smaller and grander places than Ethical AI. There is no utopia but the shelter and kindness at the hearth of those who love us.