Chapter 1: Humanity’s Special Power

Are you saying machines will become conscious?

Not necessarily, and this seems like a separate topic.

If Anyone Builds It, Everyone Dies doesn’t discuss machine consciousness at all, focusing instead on machine intelligence. As a first step in talking about consciousness, we should first clarify what kind of “consciousness” we have in mind.

When someone asks “Is my pet dog conscious?”, they can mean several different things, like:

• Does Rover really understand stuff, or is he just executing complicated instincts? Is he thinking, or just going through the motions?
• Is he self-aware? Does he know about his own existence? Can he reflect on his own thought process and build complicated mental models of himself?
• Does he have genuine subjective experiences? Does he have his own internal point of view, or is he just a mindless automaton? Is there something it’s like to be my dog? When I’m gone for a while, he yowls like he misses me; is that because he’s actually experiencing loneliness (or something along those lines)? Or is he more like a simple unconscious computer program that just exhibits the relevant behaviors without really feeling it?

We can ask similar questions about AIs.

• Does ChatGPT have “real understanding”? Well, it’s able to perform some very complex cognitive tasks very well, and others not so well. It performs well on many novel tasks that it never encountered in training — tasks that require creatively synthesizing and modifying information in new ways. So at some point, the question of whether it “really understands” starts to feel like quibbling over definitions. The practically important question — the question that’s more relevant to our survival — is what real-world capabilities AIs have now, and what capabilities they’re likely to exhibit in the coming months and years.
• Is ChatGPT self-aware? Again, ChatGPT seems good at modeling itself in some ways, and bad at modeling itself in other ways. There’s a serious confounder in that the entire paradigm that led to ChatGPT was focused on making things that sound like they’re self-aware by giving the same sorts of responses that humans would give. People can argue about whether ChatGPT has crossed some important thresholds in self-awareness, and they can argue about what thresholds lie in the future. But sooner or later, we can expect AIs to exist that have extremely powerful practical abilities to understand and reason about themselves — the ability to debug themselves, to design new and improved versions of themselves, to make complicated plans about their position in the world, etc.
• Does ChatGPT have genuine subjective experiences?

The last of these questions is the most philosophically thorny, and leads to a cluster of questions surrounding whether AIs like ChatGPT are entities worth moral concern. We will discuss those topics later, once we have covered a few more background concepts.

When we use the word “conscious,” we’re specifically thinking of “having subjective experience” and not things like self-modeling and deep practical understanding.^[1]

Our best guess is that AIs today are probably not conscious (although we’re more uncertain every year), and that subjective experience isn’t necessary for superintelligence.

But these are just guesses, albeit ones based on a reasonable amount of thinking and theorizing. We don’t think it’s at all silly to worry that some current or future AI systems might be conscious, or even to worry about whether we might be badly mistreating current AIs, especially when they do things like threaten to kill themselves after failing to debug code.

Any entity that would constitute a superintelligence by our lights would necessarily be extremely good at modeling itself — thinking about its own computations, improving its mental heuristics, understanding and predicting the impacts of its own behavior on the surrounding environment, etc. But our best guess is that human-style self-aware consciousness is just one particular way that a mind can effectively model itself; it’s not a necessary prerequisite for reflective reasoning.

Consciousness may be an important part of how humans are so good at manipulating the world, but that doesn’t mean that non-conscious machines would be defective and unable to predict and steer the world. Submarines don’t swim in a fashion analogous to humans; they accomplish the task of moving through the water in a fundamentally different way. We expect an AI to be able to succeed at the same challenges that humans succeed at, but not necessarily to do them via the same subjective experience channel that humans use.

(See also the analogous case of curiosity, which we’ll turn to in the supplement to Chapter 4.)

To put it another way: Blood is very important in the operation of a human arm, but that doesn’t mean that robot arms require blood to operate. A robotic arm is not defective in the way that a bloodless human arm would be; it just works in a different, bloodless way. Our best guess is that machine superintelligences will work in a different, non-conscious way — although this guess is not important to our argument in the book.

Our focus in If Anyone Builds It, Everyone Dies is on intelligence — where “intelligence” is defined in terms of a reasoner’s ability to predict and steer the world, regardless of whether that reasoner’s brain works like a human brain. If an AI is inventing new technology and infrastructure and proliferating it across the face of the planet in a fashion that kills us as a side effect, then stopping to ask “But is it conscious?” seems somewhat academic.

We’ll go into more detail on why we think prediction and steering probably don’t require consciousness (and what this means for how we should think about AI welfare and AI rights) after Chapter 5, once we’ve laid more groundwork. See “Effectiveness, Consciousness, and AI Welfare” for that discussion.

Notes

[1] threaten to kill themselves: Users report that Google’s Gemini AI threatens to uninstall itself from projects, or to delete all its code, when it’s having trouble completing some task.