Philosopher Nick Bostrom recently posted a paper, where he postulated that a small chance of AI annihilating all humans might be worth the risk, because advanced AI might relieve humanity of “its universal death sentence.” That upbeat gamble is quite a leap from his previous dark musings on AI, which made him a doomer godfather. His 2014 book Superintelligence was an early examination of AI’s existential risk. One memorable thought experiment: An AI tasked with making paper clips winds up destroying humanity because all those resource-needy people are an impediment to paper clip production. His more recent book, Deep Utopia, reflects a shift in his focus. Bostrom, who leads Oxford’s Future of Humanity Institute, dwells on the “solved world” that comes if we get AI right.

STEVEN LEVY: Deep Utopia is more optimistic than your previous book. What changed for you?

NICK BOSTROM: I call myself a fretful optimist. I am very excited about the potential for radically improving human life and unlocking possibilities for our civilization. That’s consistent with the real possibility of things going wrong.

You wrote a paper with a striking argument: Since we’re all going to die anyway, the worst that can happen with AI is that we die sooner. But if AI works out, it might extend our lives, maybe indefinitely.

That paper explicitly looks at only one aspect of this. In any given academic paper, you can't address life, the universe, and the meaning of everything. So let's just look at this little issue and try to nail that down.

That isn’t a little issue.

I guess I've been irked by some of the arguments made by doomers who say that if you build AI, you're going to kill me and my children and how dare you. Like the recent book If Anyone Builds It, Everyone Dies. Even more probable is that if nobody builds it, everyone dies! That's been the experience for the last several 100,000 years.

But in the doomer scenario everybody dies and there’s no more people being born. Big difference.

I have obviously been very concerned with that. But in this paper, I'm looking at a different question, which is, what would be best for the currently existing human population like you and me and our families and the people in Bangladesh? It does seem like our life expectancy would go up if we develop AI, even if it is quite risky.

In Deep Utopia you speculate that AI could create incredible abundance, so much that humanity might have a huge problem with finding purpose. I live in the United States. We're a very rich country, but our government, ostensibly with support of the people, has policies that deny services to the poor and distribute rewards to the rich. I think that even if AI was able to provide abundance for everyone, we would not supply it to everyone.

You might be right. Deep Utopia takes as its starting point the postulation that everything goes extremely well. If we do a reasonably good job on governance, everybody gets a share. There is quite a deep philosophical question of what a good human life would look like under these ideal circumstances.

The meaning of life is something you hear a lot about in Woody Allen movies and maybe in the philosophers community. I’m worried more about the wherewithal to support oneself and get a stake in this abundance.

The book is not only about meaning. That’s one out of a bunch of different values that it considers. This could be a wonderful emancipation from the drudgery that humans have been subjected to. If you have to give up, say, half of your waking hours as an adult just to make ends meet, doing some work you don't enjoy and that you don't believe in, that’s a sad condition. Society is so used to it that we've invented all kinds of rationalizations around it. It’s like a partial form of slavery.

When the moment comes when AI writes philosophy papers better than you do, will some meaning be drained from your existence?

I think so. The ability to make some big contribution to the world, or help save the world, or ensure the future will be out of my hands, and maybe out of everybody's hands.

On the other hand, a philosophy paper written by a human could be more valuable than a much cleverer, deeper philosophy paper written by a nonhuman, because I'm a human and that relates to me.

I guess you could have philosophy as kind of a sport.

That's not just sport. The proclamations of a robot aren’t as meaningful to me as those of a fellow human.

I guess it's the same if you retire after a career you're passionate about and feel you're good at. Maybe you have a great retirement, and you enjoy relaxing and reading the books you have time for, and playing with your grandkids, but there's still something probably that you might miss, that you feel is lost. Maybe this will be analogous to a big retirement for humanity, but hopefully a retirement of enormous vitality. These utopians living in the solved world would be doing things like games and aesthetic, spiritual, and religious activities.

If you were in charge of one of the hyperscalers, what would you do differently than what they're doing now?

A bigger effort should be done on the welfare of digital minds. Anthropic has been a pioneer there. It’s not clear that current AIs have moral status yet, but starting the process brings us into a mindset as a civilization to do more as these systems become sophisticated. It’s very plausible that some of these digital minds that we're constructing will have various degrees of moral status, just as we think pigs and dogs have moral status. If you kick somebody's dog, maybe you harm the owner, but it's also bad because it hurts the dog. If AIs have a conception of self as existing through time and life, goals that they want to achieve, and the ability to form reciprocal relationships with other beings and humans, then I think there would be ways of treating them that would be wrong.

In your book you say maybe we shouldn’t treat “digital minds” as if they were animals in factory farming. I’m worried about whether they might make us the animals in some equivalent of factory farming.

Hence the importance of the alignment problem. We are not just waiting for these AI super-beings to come into existence and hoping that they will be friendly, we get to shape them and raise them. That gives us an opportunity to increase the chances that they will have some affinity for us.

If AIs have goals that run counter to ours, wouldn’t that be a failure to align them with human values?

If we fail to solve the alignment, as we probably will do at least to some degree, it's important that they can be accommodated and given a good future. There are a lot of win-win opportunities that arise if we approach them not merely as objects to be exploited to the maximum degree, but try to foster a positive relationship. The most important relationship, ultimately, might be the one between humans and AIs. So it would be more promising that the relationship goes well if we start by taking some steps towards being generous and kind and respectful.

[This interview was edited for length and coherence.]

This is an edition of Steven Levy’s Backchannel newsletter. Read previous newsletters here.