Chapter 10
A Cursed Problem
ASI alignment is the challenge of getting useful work from an artificial superintelligence (ASI), reliably and without causing a catastrophe. This looks like a very difficult challenge because of various basic features of the problem.
Below we cover follow-up questions for people who have read Chapter 10 of If Anyone Builds It, Everyone Dies. We’ll go into more detail about how informative various historical comparisons are, and we’ll consider proposals for scenarios that might make the problem easier. Topics we won’t cover here, to avoid rehashing the book, include:
- What makes an engineering problem hard?
- What kinds of hard problems has humanity faced in its history, and what lessons can we learn from those as we think about the path to ASI?
- If you know up front that you are facing a hard problem, what can you do? How should one behave differently when facing a truly hard problem?