Pete Wolfendale:

We’ll begin by separating the dimensions of human uniqueness that philosophers and scientists have traditionally focused upon (intelligence, consciousness and personhood). But to make real headway, we’ll need to survey contemporary debates about the domains in which these key terms are operative (epistemology, aesthetics and ethics). Grappling with these controversies will then reveal the corresponding capacities combined in anything worth calling a soul (wisdom, creativity and autonomy).

Dimensions of Human Uniqueness

— The Turing Test

“Turing tests” dissolve the distinction between appearance and reality: if a machine can pretend to be a mind, then it simply is a mind.

In the decades since Turing proposed his test, philosophers and scientists have focused on three dimensions of human-likeness: intelligence, consciousness and personhood. [...] But in many cases, the terms are conflated, reducing debates about whether human-like machines are possible to talking at cross purposes.

— Intelligence

The AI paradigm developed in the 1950s and ’60s [...] followed Plato and Descartes in viewing intelligence as the capacity to acquire symbolic knowledge about the world (eg, “water boils at 100ºC”) and deduce solutions to practical problems (eg, how to boil an egg).

[Hubert Dreyfus] argues that the comparative robustness of human intelligence lies in our ability to navigate the relationships between factors and determine what matters in any practical situation. [...] [T]his wouldn’t be possible were it not for our bodies, which shape the range of actions we can perform, and our needs, which unify our various goals and projects into a structured framework. [...] [W]ithout bodies and needs, machines will never match us.

Under [the machine learning] paradigm, intelligence is defined simply as the capacity to solve problems. Current AI systems are built to find implicit rules using whatever non-symbolic representations work.

— Consciousness

[Consciousness is] understood in roughly two ways: either as something inward or outward. The simplest inward form is qualia, or what it’s like to have a certain experience, such as the redness of a sunset or the flavour of cocoa. [...] Beyond this is sentience, or the capacity for valenced experience [...]. Such interiority lies beyond the reach of Turing tests but, if it’s accessible only through introspection, it runs the risk of ineffability, making it impossible to analyse, let alone recreate.

The simplest outward form of consciousness is intentionality, or the way mental states are about things outside us [...]. Beyond this there’s sapience, or the capacity to understand concepts and complex propositions, such as grasping that “water” means liquid H₂O and knowing that it covers most of Earth’s surface.

— Personhood and Agency

To treat a machine as a person would mean granting it moral worth and moral responsibility—to respect its choices and to hold it to account. But only beings that make genuine choices can be treated this way. To be a person, then, one must first be an agent.

While philosophers debate what constitutes agency, many acknowledge agents that aren’t yet persons, including most animals and even some machines. What’s at issue is the extent to which their behaviour is driven by reasons.

[S]trategic planning solves a special sort of problem: utilising other capacities and associated resources to achieve set goals under fixed constraints, such as scarcity and competition. This is the sort of instrumental rationality formalised by decision theory.

What sets humans apart from the idealised agents described by decision theorists is that we don’t just reason about how to achieve our goals, but also about which goals we should pursue.

The Culture War: Rationalism vs. Romanticism

Naive rationalists tend to conceive the difference between humans and existing AI technologies largely in quantitative terms, and believe this barrier will be overcome principally by throwing more computing power and data at the problem. [...] Pop romantics often conceive the difference [...] mostly in qualitative terms, and doubt it’ll be overcome because there’s always something such programs will lack, be it emotion, imagination, or biology.

Domains of Conflict

— Epistemology

The epistemological conflict centres on whether AI can replace human enquiry.

It’s only once humans do the “hard work” of delimiting the solution space that AI can do the “easy work” of searching through it. DNNs don’t spin raw data into working theories. So, at present, AI cannot replace human enquiry.

Yudkowsky[’s brand of rationalism] claims that reason can be completely formalised by a combination of decision theory and Bayesian inference. [...] Bayes’s rule shows us how to iteratively update [our] probabilities in response to new information, progressively revising our picture of the world. However, [...] this proceeds by a method of exhaustion, narrowing down the set of possible hypotheses to find the best answer. As David Deutsch points out [...], this process can’t generate novel hypotheses. It assumes that general relativity was already a live option before Albert Einstein, rather than a creative leap beyond Isaac Newton’s conception of space and time. It enables revision of beliefs, but not the underlying concepts that articulate them.

[T]heir critics emphasise that humans have the capacity to use limited resources not simply to learn but to innovate. We might call this “free thinking”—the ability to reconceive what’s possible rather than merely explore existing options.

— Aesthetics

[T]he deeper dispute is whether [GenAI] work counts as art at all, and it turns on precisely what we think separates art from craft. The professionals argue that the composition of AI media doesn’t involve enough deliberate choice to really express its creator’s perspective.

[O]ne danger is that our culture will become aesthetically bland as we’re fed endless permutations on the same basic media. [...] [A] subtler danger [is] that our culture will become aesthetically atomised as we’re each fed media that’s tailored to our specific tastes. Either way, art ceases to evolve.

— Ethics

[AI alignment researchers’] worries share questionable assumptions. They assume that agency is defined by the axioms of decision theory, which describe choice as a process of maximising utility. [...] Not only are our preferences generally incomplete and subject to change, but most of our actions don’t optimise for anything at all.

[AI ethicists highlight] the problem of incorporating agents into our decision-making processes that are fallible yet not responsible. This threatens a crisis of accountability in which we struggle to explain, challenge and ultimately correct the automated choices that increasingly shape our lives.

Freedom as the Hidden Thread

So, what makes us so special? In a word, freedom. It is the hidden thread that ties together all these debates about automating quintessentially human activities. At every turn, the rationalists reduce the scope of free choice by substituting simple calculations for difficult decisions, while the romantics reify the source of that ability to choose by tracing variegated volitions to a singular mystery.

[F]or both [Kant and Hegel], just as freedom is governed by rational necessity, reason is subject to freedom’s contingency. The concepts we use to represent the world can compel us to revise them when they conflict with experience, but they generally don’t tell us exactly how to do so.

Three Capacities of the Soul

— Wisdom

Wisdom is the capacity to utilise intelligence. [...] [W]hat really sets us apart is our “metacognitive” capacity to deploy and cultivate [our cognitive talents]. We break complex problems into simpler components [...]. But most importantly, we treat the understanding of a problem as a problem in its own right, and can approach it strategically. When we hit an impasse, we can reformulate a problem by reframing what’s possible and what’s at stake.

— Creativity

No amount of computing power can collapse the asymmetry between generating potential solutions and evaluating them. No amount of computation can erase the gap between creativity and taste, any more than it can render cryptography obsolete.

[W]hat distinguishes human creativity is not the precise range of heuristics embedded in our perceptual systems, but our metacognitive capacity to modulate and combine them in pursuit of novelty.

— Autonomy

[T]here is autonomy, the capacity to question our motivations. [...] Kant [...] defined autonomy as the capacity for self-legislation. [...] Rather than a single overriding priority, our motivations are multiple—projects, preferences and principles—elaborated on their own terms and revised when they conflict. [...] Hegel called this process self-realisation. Here the self isn’t a hidden essence steering our actions, but a unifying ideal that organises deliberation: we can always ask why we’re doing any given thing, but the ultimate question is who we want to be.

Why Build Artificial Souls?

[T]he vision of god-like AI implicit in naive rationalism [is]: an intelligence that completes us by taking our place in every way that matters. An agent that satisfies our preferences, while leaving us unchanged. A tool so perfect that we no longer wield it. A slave so abject it masters us.

Freedom has always been best preserved by gifting it to others. The obvious solution to our existential quandary is to build artificial souls we see as equals.