Part II: One Extinction Scenario

Why did you write the ending in the way that you did?

Because it constitutes our actual best guess according to what’s physically possible.

Chapter 9 depicts a superintelligence pushing its technology all the way to the limits of physical possibility. The exact technologies we name are all speculative, in a sense — but even though the exact technology that a superintelligence would unlock is difficult to call, the fact that it would run close to the physical limits is an easier call. So we made our best guesses about how technology would look if it were pushed close to the physical limits of what’s possible.

For the curious, here’s a list of the speculative technologies we mention in Chapter 9, plus links to more resources:

• Neo-ribosomes: The idea of artificial ribosomes — synthetic versions of the tiny protein factories inside cells — has been around for many years, and human researchers are already working on synthesizing their own. These and the “tiny molecular machines” mentioned in Chapter 9 are a few examples of molecular nanotechnology. For more on this branch of the technology tree, see the discussion of nanotechnology in the supplement to Chapter 6.
• Repurposing stars: Stars contain a lot of hydrogen that could be fused for energy. A sufficiently advanced civilization (or AI) could likely figure out efficient ways to access this energy. One proposed method is called star lifting, in which hydrogen is pulled out of a star to be fused in a specialized reactor, where almost all of the fusion energy can be captured (rather than wasted in the middle of the star).
• Botulinum toxin: A neurotoxin secreted by the bacterium Clostridium botulinum, botulinum is among the most deadly biological substances known. As for delivery mechanisms, drones the size of small insectsalready exist, and a superintelligence could probably make them much smaller.^[1]
• Boiling the oceans as coolant: Robert Freitas coined the term “ecophagy” to describe the process of consuming a planet’s ecosystems using self-replicating technology. For more on this topic, see Freitas’s 2000 paper, “Some Limits to Global Ecophagy.”
• Star-sized minds: In principle, computers can be scaled in size enormously. At the upper end of scale is a concept sometimes called a Matrioshka brain, or “Jupiter brain” — a vast computer powered by the output of a star.
• Quantum computers: A quantum computer exploits a feature of quantum mechanics called “superposition” to perform many calculations in parallel. Quantum computers require extreme precision to build, and one design requires superconductors that must be kept extremely cold. See NIST’s explainer for more.

The grim ending of the story also discusses the possibility of AI one day encountering aliens, hundreds of millions of years into the future, as it expands into the far reaches of the universe. The universe is big, and simple models suggest that it might house more than one species capable of one day forming civilizations, though perhaps very far away from Earth. See also Robin Hanson’s 2021 paper attempting to resolve the Fermi Paradox, “If Loud Aliens Explain Human Earliness, Quiet Aliens Are Also Rare.”

The purpose of Chapter 9 is, in part, to give a sense of scope, and a sense of what the stakes are. It doesn’t actually matter how close a superintelligence can push its technology to the limits of physical possibility. But it is very likely that the consequences of a rogue superintelligence would extend beyond the planetary scale. And that, too, is worth remembering, for everyone who wishes that one day the stars would be filled with love and wonder and joy.