[draft] Responses to Catastrophic AGI Risk: A Survey

Here’s the biggest thing that I’ve been working on for the last several months:

Responses to Catastrophic AGI Risk: A SurveyKaj Sotala, Roman Yampolskiy, and Luke Muehlhauser

*Abstract: *Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next 20-100 years. It has been suggested that this may become a catastrophic risk, threatening to do major damage on a global scale. After briefly summarizing the arguments for why AGI may become a catastrophic risk, we survey various proposed responses to AGI risk. We consider societal proposals, proposals for constraining the AGIs’ behavior from the outside, and for creating AGIs in such a way that they are inherently safe.

This doesn’t aim to be a very strongly argumentative paper, though it does comment on the various proposals from an SI-ish point of view. Rather, it attempts to provide a survey of all the major AGI-risk related proposals that have been made so far, and to provide some thoughts on their respective strengths and weaknesses. Before writing this paper, we hadn’t encountered anyone who’d have been familiar with all of these proposals—not to mention that even we ourselves weren’t familiar with all of them! Hopefully, this should become a useful starting point for anyone who’s at all interested in AGI risk or Friendly AI.

The draft will be public and open for comments for one week (until Nov 23rd), after which we’ll incorporate the final edits and send it off for review. We’re currently aiming to have it published in the sequel volume to Singularity Hypotheses.

EDIT: I’ve now hidden the draft from public view (so as to avoid annoying future publishers who may not like early drafts floating around before the work has been accepted for publication) while I’m incorporating all the feedback that we got. Thanks to everyone who commented!