Relevant pre-AGI possibilities

Link post Contents

• Interactive "Generate Future" button
• Key
• List of strategically relevant possibilities
• Inputs to AI
• Technical tools
• Human effectiveness
• Attitudes toward AGI
• Precursors to AGI
• Shifts in the balance of power
• Notes Epistemic status: I started this as an AI Impacts research project, but given that it’s fundamentally a fun speculative brainstorm, it worked better as a blog post. The default, when reasoning about advanced artificial general intelligence (AGI), is to imagine it appearing in a world that is basically like the present. Yet almost everyone agrees the world will likely be importantly different by the time advanced AGI arrives. One way to address this problem is to reason in abstract, general ways that are hopefully robust to whatever unforeseen developments lie ahead. Another is to brainstorm particular changes that might happen, and check our reasoning against the resulting list. This is an attempt to begin the second approach.2 I sought things that might happen that seemed both (a) within the realm of plausibility, and (b) probably strategically relevant to AI safety or AI policy. I collected potential list entries via brainstorming, asking others for ideas, googling, and reading lists that seemed relevant (e.g. Wikipedia’s list of emerging technologies,3 a list of Ray Kurzweil’s predictions4, and DARPA’s list of projects.5) I then shortened the list based on my guesses about the plausibility and relevance of these possibilities. I did not put much time into evaluating any particular possibility, so my guesses should not be treated as anything more. I erred on the side of inclusion, so the entries in this list vary greatly in plausibility and relevance. I made some attempt to categorize these entries and merge similar ones, but this document is fundamentally a brainstorm, not a taxonomy, so keep your expectations low. I hope to update this post as new ideas find me and old ideas are refined or refuted. I welcome suggestions and criticisms; email me (gmail kokotajlod) or leave a comment.

Interactive "Generate Future" button

Asya Bergal and I made an interactive button to go with the list. The button randomly generates a possible future according to probabilities that you choose. It is very crude, but it has been fun to play with, and perhaps even slightly useful. For example, once I decided that my credences were probably systematically too high because the futures generated with them were too crazy. Another time I used the alternate method (described below) to recursively generate a detailed future trajectory, written up here. I hope to make more trajectories like this in the future, since I think this method is less biased than the usual method for imagining detailed futures.6 To choose probabilities, scroll down to the list below and fill each box with a number representing how likely you think the entry is to occur in a strategically relevant way prior to the advent of advanced AI. (1 means certainly, 0 means certainly not. The boxes are all 0 by default.) Once you are done, scroll back up and click the button. A major limitation is that the button doesn’t take correlations between possibilities into account. The user needs to do this themselves, e.g. by redoing any generated future that seems silly, or by flipping a coin to choose between two generated possibilities that seem contradictory, or by choosing between them based on what else was generated. Here is an alternate way to use this button that mostly avoids this limitation:

• Fill all the boxes with probability-of-happening-in-the-next-5-years (instead of happening before advanced AGI, as in the default method)

• Click the "Generate Future" button and record the results, interpreted as what happens in the next 5 years.

• Update the probabilities accordingly to represent the upcoming 5-year period, in light of what has happened so far.

• Repeat steps 2 – 4 until satisfied. I used a random number generator to determine whether AGI arrived each year. If you don’t want to choose probabilities yourself, click "fill with pre-set values" to populate the fields with my non-expert, hasty guesses.7 GENERATE FUTUREFill with pre-set values (default method)Fill with pre-set values (alternate method)

Key

Letters after list titles indicate that I think the change might be relevant to:

• TML: Timelines—how long it takes for advanced AI to be developed

• TAS: Technical AI safety—how easy it is (on a technical level) to make advanced AI safe, or what sort of technical research needs to be done

• POL: Policy—how easy it is to coordinate relevant actors to mitigate risks from AI, and what policies are relevant to this.

• CHA: Chaos—how chaotic the world is.8

• MIS: Miscellaneous Each possibility is followed by some explanation or justification where necessary, and a non-exhaustive list of ways the possibility may be relevant to AI outcomes in particular (which is not guaranteed to cover the most important ones). Possibilities are organized into loose categories created after the list was generated.

List of strategically relevant possibilities

Inputs to AI****1. Advanced science automation and research tools (TML, TAS, CHA, MIS) Narrow research and development tools might speed up technological progress in general or in specific domains. For example, several of the other technologies on this list might be achieved with the help of narrow research and development tools. 2. Dramatically improved computing hardware (TML, TAS, POL, MIS) By this I mean computing hardware improves at least as fast as Moore’s Law. Computing hardware has historically become steadily cheaper, though it is unclear whether this trend will continue. Some example pathways by which hardware might improve at least moderately include:

• Ordinary scale economies9

• Improved data locality10

• Increased specialization for specific AI applications11

• Optical computing12

• Neuromorphic chips13

• 3D integrated circuits14

• Wafer-scale chips15

• Quantum computing16

• Carbon nanotube field-effect transistors17 Dramatically improved computing hardware may:

• Cause any given AI capability to arrive earlier

• Increase the probability of hardware overhang.

• Affect which kinds of AI are developed first (e.g. those which are more compute-intensive.)

• Affect AI policy, e.g. by changing the relative importance of hardware vs. research talent 3. Stagnation in computing hardware progress (TML, TAS, POL, MIS) Many forecasters think Moore’s Law will be ending soon (as of 2020).18 In the absence of successful new technologies, computing hardware could progress substantially more slowly than Moore’s Law would predict. Stagnation in computing hardware progress may:

• Cause any given AI capability to arrive later

• Decrease the probability of hardware overhang.

• Affect which kinds of AI are developed first (e.g. those which are less compute-intensive.)

• Influence the relative strategic importance of hardware compared to researchers

• Make energy and raw materials a greater part of the cost of computing 4. Manufacturing consolidation (POL) Chip fabrication has become more specialized and consolidated over time, to the point where all of the hardware relevant to AI research depends on production from a handful of locations.19 Perhaps this trend will continue. One country (or a small number working together) could control or restrict AI research by controlling the production and distribution of necessary hardware. 5. Advanced additive manufacturing (e.g. 3D printing or nanotechnology) (TML, CHA) Advanced additive manufacturing could lead to various materials, products and forms of capital being cheaper and more broadly accessible, as well as to new varieties of them becoming feasible and quicker to develop. For example, sufficiently advanced 3D printing could destabilize the world by allowing almost anyone to secretly produce terror weapons. If nanotechnology advances rapidly, so that nanofactories can be created, the consequences could be dramatic:20

• Greatly reduced cost of most manufactured products

• Greatly faster growth of capital formation

• Lower energy costs

• New kinds of materials, such as stronger, lighter spaceship hulls

• Medical nanorobots

• New kinds of weaponry and other disruptive technologies 6. Massive resource glut (TML, TAS, POL, CHA) By "glut" I don’t necessarily mean that there is too much of a resource. Rather, I mean that the real price falls dramatically. Rapid decreases in the price of important resources have happened before.21 It could happen again via:

• Cheap energy (e.g. fusion power, He-3 extracted from lunar regolith,22 methane hydrate extracted from the seafloor,23 cheap solar energy24)

• A source of abundant cheap raw materials (e.g. asteroid mining,25 undersea mining26)

• Automation of relevant human labor. Where human labor is an important part of the cost of manufacturing, resource extraction, or energy production, automating labor might substantially increase economic growth, which might result in a greater amount of resources devoted to strategically relevant things (such as AI research) which is relevantly similar to a price drop even if technically the price doesn’t drop.27 and therefore investment in AI. My impression is that energy, raw materials, and unskilled labor combined are less than half the cost of computing, so a decrease in the price of one of these (and possibly even all three) would probably not have large direct consequences on the price of computing.28 But a resource glut might lead to general economic prosperity, with many subsequent effects on society, and moreover the cost structure of computing may change in the future, creating a situation where a resource glut could dramatically lower the cost of computing.29 7. Hardware overhang (TML, TAS, POL) Hardware overhang refers to a situation where large quantities of computing hardware can be diverted to running powerful AI systems as soon as the AI software is developed. If advanced AGI (or some other powerful software) appears during a period of hardware overhang, its capabilities and prominence in the world could grow very quickly. 8. Hardware underhang (TML, TAS, POL) The opposite of hardware overhang might happen. Researchers may understand how to build advanced AGI at a time when the requisite hardware is not yet available. For example, perhaps the relevant AI research will involve expensive chips custom-built for the particular AI architecture being trained. A successful AI project during a period of hardware underhang would not be able to instantly copy the AI to many other devices, nor would they be able to iterate quickly and make an architecturally improved version. Technical tools****9. Prediction tools (TML, TAS, POL, CHA, MIS) Tools may be developed that are dramatically better at predicting some important aspect of the world; for example, technological progress, cultural shifts, or the outcomes of elections, military clashes, or research projects. Such tools could for instance be based on advances in AI or other algorithms, prediction markets, or improved scientific understanding of forecasting (e.g. lessons from the Good Judgment Project). Such tools might conceivably increase stability via promoting accurate beliefs, reducing surprises, errors or unnecessary conflicts. However they could also conceivably promote instability via conflict encouraged by a powerful new tool being available to a subset of actors. Such tools might also help with forecasting the arrival and effects of advanced AGI, thereby helping guide policy and AI safety work. They might also accelerate timelines, for instance by assisting project management in general and notifying potential investors when advanced AGI is within reach. 10. Persuasion tools (POL, CHA, MIS) Present technology for influencing a person’s beliefs and behavior is crude and weak, relative to what one can imagine. Tools may be developed that more reliably steer a person’s opinion and are not so vulnerable to the victim’s reasoning and possession of evidence. These could involve:

• Advanced understanding of how humans respond to stimuli depending on context, based on massive amounts of data

• Coaching for the user on how to convince the target of something

• Software that interacts directly with other people, e.g. via text or email Strong persuasion tools could:

• Allow a group in conflict who has them to quickly attract spies and then infiltrate an enemy group

• Allow governments to control their populations

• Allow corporations to control their employees

• Lead to a breakdown of collective epistemology30 11. Theorem provers (TAS) Powerful theorem provers might help with the kinds of AI alignment research that involve proofs or help solve computational choice problems. 12. Narrow AI for natural language processing (TML, TAS, CHA) Researchers may develop narrow AI that understands human language well, including concepts such as "moral" and "honest." Natural language processing tools could help with many kinds of technology, including AI and various AI safety projects. They could also help enable AI arbitration systems. If researchers develop software that can autocomplete code—much as it currently autocompletes text messages—it could multiply software engineering productivity. 13. AI interpretability tools (TML, TAS, POL) Tools for understanding what a given AI system is thinking, what it wants, and what it is planning would be useful for AI safety.31 14. Credible commitment mechanisms (POL, CHA) There are significant restrictions on which contracts governments are willing and able to enforce–for example, they can’t enforce a contract to try hard to achieve a goal, and won’t enforce a contract to commit a crime. Perhaps some technology (e.g. lie detectors, narrow AI, or blockchain) could significantly expand the space of possible credible commitments for some relevant actors: corporations, decentralized autonomous organizations, crowds of ordinary people using assurance contracts, terrorist cells, rogue AGIs, or even individuals. This might destabilize the world by making threats of various kinds more credible, for various actors. It might stabilize the world in other ways, e.g. by making it easier for some parties to enforce agreements. 15. Better coordination tools (POL, CHA, MIS) Technology for allowing groups of people to coordinate effectively could improve, potentially avoiding losses from collective choice problems, helping existing large groups (e.g. nations and companies) to make choices in their own interests, and producing new forms of coordinated social behavior (e.g. the 2010’s saw the rise of the Facebook group)). Dominant assurance contracts,32 improved voting systems,33 AI arbitration systems, lie detectors, and similar things not yet imagined might significantly improve the effectiveness of some groups of people. If only a few groups use this technology, they might have outsized influence. If most groups do, there could be a general reduction in conflict and increase in good judgment. Human effectiveness****16. Deterioration of collective epistemology (TML, TAS, POL, CHA, MIS) Society has mechanisms and processes that allow it to identify new problems, discuss them, and arrive at the truth and/​or coordinate a solution. These processes might deteriorate. Some examples of things which might contribute to this:

• Increased investment in online propaganda by more powerful actors, perhaps assisted by chatbots, deepfakes and persuasion tools

• Echo chambers, filter bubbles, and online polarization, perhaps driven in part by recommendation algorithms

• Memetic evolution in general might intensify, increasing the spreadability of ideas/​topics at the expense of their truth/​importance34

• Trends towards political polarization and radicalization might exist and continue

• Trends towards general institutional dysfunction might exist and continue This could cause chaos in the world in general, and lead to many hard-to-predict effects. It would likely make the market for influencing the course of AI development less efficient (see section on "Landscape of…" below) and present epistemic hazards for anyone trying to participate effectively. 17. New and powerful forms of addiction (TML, POL, CHA, MIS) Technology that wastes time and ruins lives could become more effective. The average person spends 144 minutes per day on social media, and there is a clear upward trend in this metric.35 The average time spent watching TV is even greater.36 Perhaps this time is not wasted but rather serves some important recuperative, educational, or other function. Or perhaps not; perhaps instead the effect of social media on society is like the effect of a new addictive drug — opium, heroin, cocaine, etc. — which causes serious damage until society adapts. Maybe there will be more things like this: extremely addictive video games, or newly invented drugs, or wireheading (directly stimulating the reward circuitry of the brain).37 This could lead to economic and scientific slowdown. It could also concentrate power and influence in fewer people—those who for whatever reason remain relatively unaffected by the various productivity-draining technologies. Depending on how these practices spread, they might affect some communities more or sooner than others. 18. Medicine to boost human mental abilities (TML, CHA, MIS) To my knowledge, existing "study drugs" such as modafinil don’t seem to have substantially sped up the rate of scientific progress in any field. However, new drugs (or other treatments) might be more effective. Moreover, in some fields, researchers typically do their best work at a certain age. Medicine which extends this period of peak mental ability might have a similar effect. This could speed up the rate of scientific progress in some fields, among other effects. 19. Genetic engineering, human cloning, iterated embryo selection (TML, POL, CHA, MIS) Changes in human capabilities or other human traits via genetic interventions38 could affect many areas of life. If the changes were dramatic, they might have a large impact even if only a small fraction of humanity were altered by them. Changes in human capabilities or other human traits via genetic interventions might:

• Accelerate research in general

• Differentially accelerate research projects that depend more on "genius" and less on money or experience

• Influence politics and ideology

• Cause social upheaval

• Increase the number of people capable of causing great harm

• Have a huge variety of effects not considered here, given the ubiquitous relevance of human nature to events

• Shift the landscape of effective strategies for influencing AI development (see below) 20. Landscape of effective strategies for influencing AI development changes substantially (CHA, MIS) For a person at a time, there is a landscape of strategies for influencing the world, and in particular for influencing AI development and the effects of advanced AGI. The landscape could change such that the most effective strategies for influencing AI development are:

• More or less reliably helpful (e.g. working for an hour on a major unsolved technical problem might have a low chance of a very high payoff, and so not be very reliable)

• More or less "outside the box" (e.g. being an employee, publishing academic papers, and signing petitions are normal strategies, whereas writing Harry Potter fanfiction to illustrate rationality concepts and inspire teenagers to work on AI safety is not)39

• Easier or harder to find, such that marginal returns to investment in strategy research change Here is a non-exhaustive list of reasons to think these features might change systematically over time:

• As more people devote more effort to achieving some goal, one might expect that effective strategies become common, and it becomes harder to find novel strategies that perform better than common strategies. As advanced AI becomes closer, one might expect more effort to flow into influencing the situation. Currently some ‘markets’ are more efficient than others; in some the orthodox strategies are best or close to the best, whereas in others clever and careful reasoning can find strategies vastly better than what most people do. How efficient a market is depends on how many people are genuinely trying to compete in it, and how accurate their beliefs are. For example, the stock market and the market for political influence are fairly efficient, because many highly-knowledgeable actors are competing. As more people take interest, the ‘market’ for influencing the course of AI may become more efficient. (This would also decrease the marginal returns to investment in strategy research, by making orthodox strategies closer to optimal.) If there is a deterioration of social epistemology (see below), the market might instead become less efficient.

• Currently there are some tasks at which the most skilled people are not much better than the average person (e.g. manual labor, voting) and others in which the distribution of effectiveness is heavy-tailed, such that a large fraction of the total influence comes from a small fraction of individuals (e.g. theoretical math, donating to politicians). The types of activity that are most useful for influencing the course of AI development may change over time in this regard, which in turn might affect the strategy landscape in all three ways described above.

• Transformative technologies can lead to new opportunities and windfalls for people who recognize them early. As more people take interest, opportunities for easy success disappear. Perhaps there will be a burst of new technologies prior to advanced AGI, creating opportunities for unorthodox or risky strategies to be very successful. A shift in the landscape of effective strategies for influencing the course of AI is relevant to anyone who wants to have an effective strategy for influencing the course of AI.40 If it is part of a more general shift in the landscape of effective strategies for other goals — e.g. winning wars, making money, influencing politics — the world could be significantly disrupted in ways that may be hard to predict. 21. Global economic collapse (TML, CHA, MIS) This might slow down research or precipitate other relevant events, such as war. 22. Scientific stagnation (TML, TAS, POL, CHA, MIS) There is some evidence that scientific progress in general might be slowing down. For example, the millennia-long trend of decreasing economic doubling time seems to have stopped around 1960.41 Meanwhile, scientific progress has arguably come from increased investment in research. Since research investment has been growing faster than the economy, it might eventually saturate and grow only as fast as the economy.42 This might slow down AI research, making the events on this list (but not the technologies) more likely to happen before advanced AGI. 23. Global catastrophe (TML, POL, CHA) Here are some examples of potential global catastrophes:

• Climate change tail risks, e.g. feedback loop of melting permafrost releasing methane43

• Major nuclear exchange

• Global pandemic

• Volcano eruption that leads to 10% reduction in global agricultural production44

• Exceptionally bad solar storm knocks out world electrical grid45

• Geoengineering project backfires or has major negative side-effects46 A global catastrophe might be expected to cause conflict and slowing of projects such as research, though it could also conceivably increase attention on projects that are useful for dealing with the problem. It seems likely to have other hard to predict effects. Attitudes toward AGI****24. Shift in level of public attention on AGI (TML, POL, CHA, MIS) The level of attention paid to AGI by the public, governments, and other relevant actors might increase (e.g. due to an impressive demonstration or a bad accident) or decrease (e.g. due to other issues drawing more attention, or evidence that AI is less dangerous or imminent). Changes in the level of attention could affect the amount of work on AI and AI safety. More attention could also lead to changes in public opinion such as panic or an AI rights movement. If the level of attention increases but AGI does not arrive soon thereafter, there might be a subsequent period of disillusionment. 25. Change in investment in AGI development (TML, TAS, POL) There could be a rush for AGI, for instance if major nations begin megaprojects to build it. Or there could be a rush away from AGI, for instance if it comes to be seen as immoral or dangerous like human cloning or nuclear rocketry. Increased investment in AGI might make advanced AGI happen sooner, with less hardware overhang and potentially less proportional investment in safety. Decreased investment might have the opposite effects. 26. New social movements or ideological shifts (TML, TAS, POL, MIS) The communities that build and regulate AI could undergo a substantial ideological shift. Historically, entire nations have been swept by radical ideologies within about a decade or so, e.g. Communism, Fascism, the Cultural Revolution, and the First Great Awakening.47 Major ideological shifts within communities smaller than nations (or within nations, but on specific topics) presumably happen more often. There might even appear powerful social movements explicitly focused on AI, for instance in opposition to it or attempting to secure legal rights and moral status for AI agents.48 Finally, there could be a general rise in extremist movements, for instance due to a symbiotic feedback effect hypothesized by some,49 which might have strategically relevant implications even if mainstream opinions do not change. Changes in public opinion on AI might change the speed of AI research, change who is doing it, change which types of AI are developed or used, and limit or alter discussion. For example, attempts to limit an AI system’s effects on the world by containing it might be seen as inhumane, as might adversarial and population-based training methods. Broader ideological change or a rise in extremisms might increase the probability of a massive crisis, revolution, civil war, or world war. 27. Harbinger of AGI (ALN, POL, MIS) Events could occur that provide compelling evidence, to at least a relevant minority of people, that advanced AGI is near. This could increase the amount of technical AI safety work and AI policy work being done, to the extent that people are sufficiently well-informed and good at forecasting. It could also enable people already doing such work to more efficiently focus their efforts on the true scenario. 28. AI alignment warning shot (ALN, POL) A convincing real-world example of AI alignment failure could occur. This could motivate more effort into mitigating AI risk and perhaps also provide useful evidence about some kinds of risks and how to avoid them. Precursors to AGI****29. Brain scanning (TML, TAS, POL, CHA, MIS) An accurate way to scan human brains at a very high resolution could be developed. Combined with a good low-level understanding of the brain (see below) and sufficient computational resources, this might enable brain emulations, a form of AGI in which the AGI is similar, mentally, to some original human. This would change the kind of technical AI safety work that would be relevant, as well as introducing new AI policy questions. It would also likely make AGI timelines easier to predict. It might influence takeoff speeds. 30. Good low-level understanding of the brain (TML, TAS, POL, CHA, MIS) To my knowledge, as of April 2020, humanity does not understand how neurons work well enough to accurately simulate the behavior of a C. Elegans worm, though all connections between its neurons have been mapped50 Ongoing progress in modeling individual neurons could change this, and perhaps ultimately allow accurate simulation of entire human brains. Combined with brain scanning (see above) and sufficient computational resources, this may enable brain emulations, a form of AGI in which the AI system is similar, mentally, to some original human. This would change the kind of AI safety work that would be relevant, as well as introducing new AI policy questions. It would also likely make the time until AGI is developed more predictable. It might influence takeoff speeds. Even if brain scanning is not possible, a good low-level understanding of the brain might speed AI development, especially of systems that are more similar to human brains. 31. Brain-machine interfaces (TML, TAS, POL, CHA, MIS) Better, safer, and cheaper methods to control computers directly with our brains may be developed. At least one project is explicitly working towards this goal.51 Strong brain-machine interfaces might:

• Accelerate research, including on AI and AI safety52

• Accelerate in vitro brain technology

• Accelerate mind-reading, lie detection, and persuasion tools

• Deteriorate collective epistemology (e.g. by contributing to wireheading or short attention spans)

• Improve collective epistemology (e.g. by improving communication abilities)

• Increase inequality in influence among people 32. In vitro brains (TML, TAS, POL, CHA) Neural tissue can be grown in a dish (or in an animal and transplanted) and connected to computers, sensors, and even actuators.53 If this tissue can be trained to perform important tasks, and the technology develops enough, it might function as a sort of artificial intelligence. Its components would not be faster than humans, but it might be cheaper or more intelligent. Meanwhile, this technology might also allow fresh neural tissue to be grafted onto existing humans, potentially serving as a cognitive enhancer.54 This might change the sorts of systems AI safety efforts should focus on. It might also automate much human labor, inspire changes in public opinion about AI research (e.g. promoting concern about the rights of AI systems), and have other effects which are hard to predict. 33. Weak AGI (TML, TAS, POL, CHA, MIS) Researchers may develop something which is a true artificial general intelligence—able to learn and perform competently all the tasks humans do—but just isn’t very good at them, at least, not as good as a skilled human. If weak AGI is faster or cheaper than humans, it might still replace humans in many jobs, potentially speeding economic or technological progress. Separately, weak AGI might provide testing opportunities for technical AI safety research. It might also change public opinion about AI, for instance inspiring a "robot rights" movement, or an anti-AI movement. 34. Expensive AGI (TML, TAS, POL, CHA, MIS) Researchers may develop something which is a true artificial general intelligence, and moreover is qualitatively more intelligent than any human, but is vastly more expensive, so that there is some substantial period of time before cheap AGI is developed. An expensive AGI might contribute to endeavors that are sufficiently valuable, such as some science and technology, and so may have a large effect on society. It might also prompt increased effort on AI or AI safety, or inspire public thought about AI that produces changes in public opinion and thus policy, e.g. regarding the rights of machines. It might also allow opportunities for trialing AI safety plans prior to very widespread use. 35. Slow AGI (TML, TAS, POL, CHA, MIS) Researchers may develop something which is a true artificial general intelligence, and moreover is qualitatively as intelligent as the smartest humans, but takes a lot longer to train and learn than today’s AI systems. Slow AGI might be easier to understand and control than other kinds of AGI, because it would train and learn more slowly, giving humans more time to react and understand it. It might produce changes in public opinion about AI. 36. Automation of human labor (TML, TAS, POL, CHA, MIS) If the pace of automation substantially increases prior to advanced AGI, there could be social upheaval and also dramatic economic growth. This might affect investment in AI. Shifts in the balance of power****37. Major leak of AI research (TML, TAS, POL, CHA) Edward Snowden defected from the NSA and made public a vast trove of information. Perhaps something similar could happen to a leading tech company or AI project. In a world where much AI progress is hoarded, such an event could accelerate timelines and make the political situation more multipolar and chaotic. 38. Shift in favor of espionage (POL, CHA, MIS) Espionage techniques might become more effective relative to counterespionage techniques. In particular:

• Quantum computing could break current encryption protocols.55

• Automated vulnerability detection56 could turn out to have an advantage over automated cyberdefense systems, at least in the years leading up to advanced AGI. More successful espionage techniques might make it impossible for any AI project to maintain a lead over other projects for any substantial period of time. Other disruptions may become more likely, such as hacking into nuclear launch facilities, or large scale cyberwarfare. 39. Shift in favor of counterespionage (POL, CHA, MIS) Counterespionage techniques might become more effective relative to espionage techniques than they are now. In particular:

• Post-quantum encryption might be secure against attack by quantum computers.57

• Automated cyberdefense systems could turn out to have an advantage over automated vulnerability detection. Ben Garfinkel and Allan Dafoe58 give reason to think the balance will ultimately shift to favor defense. Stronger counterespionage techniques might make it easier for an AI project to maintain a technological lead over the rest of the world. Cyber wars and other disruptive events could become less likely. 40. Broader or more sophisticated surveillance (POL, CHA, MIS) More extensive or more sophisticated surveillance could allow strong and selective policing of technological development. It would also have other social effects, such as making totalitarianism easier and making terrorism harder. 41. Autonomous weapons (POL, CHA) Autonomous weapons could shift the balance of power between nations, or shift the offense-defense balances resulting in more or fewer wars or terrorist attacks, or help to make totalitarian governments more stable. As a potentially early, visible and controversial use of AI, they may also especially influence public opinion on AI more broadly, e.g. prompting anti-AI sentiment. 42. Shift in importance of governments, corporations, and other groups in AI development (POL, CHA) Currently both governments and corporations are strategically relevant actors in determining the course of AI development. Perhaps governments will become more important, e.g. by nationalizing and merging AI companies. Or perhaps governments will become less important, e.g. by not paying attention to AI issues at all, or by becoming less powerful and competent generally. Perhaps some third kind of actor (such as religion, insurgency, organized crime, or special individual) will become more important, e.g. due to persuasion tools, countermeasures to surveillance, or new weapons of guerilla warfare.59 This influences AI policy by affecting which actors are relevant to how AI is developed and deployed. 43. Catastrophe in strategically important location (TML, POL, CHA, MIS) Perhaps some strategically important location (e.g. tech hub, seat of government, or chip fab) will be suddenly destroyed. Here is a non-exhaustive list of ways this could happen:

• Terrorist attack with weapon of mass destruction

• Major earthquake, flood, tsunami, etc. (e.g. this research claims a 2% chance of magnitude 8.0 or greater earthquake in San Francisco by 2044.)60 If it happens, it might be strategically disruptive, causing e.g. the dissolution and diaspora of the front-runner AI project, or making it more likely that some government makes a radical move of some sort. 44. Change in national AI research loci (POL, CHA) For instance, a new major national hub of AI research could arise, rivalling the USA and China in research output. Or either the USA or China could cease to be relevant to AI research. This might make coordinating AI policy more difficult. It might make a rush for AGI more or less likely. 45. Large war (TML, POL, CHA, MIS) This might cause short-term, militarily relevant AI capabilities research to be prioritized over AI safety and foundational research. It could also make global coordination on AI policy difficult. 46. Civil war or regime change in major relevant countries (POL, CHA, MIS) This might be very dangerous for people living in those countries. It might change who the strategically relevant actors are for shaping AI development. It might result in increased instability, or cause a new social movement or ideological shift. 47. Formation of a world government (POL, CHA) This would make coordinating AI policy easier in some ways (e.g. there would be no need for multiple governing bodies to coordinate their policy at the highest level), however it might be harder in others (e.g. there might be a more complicated regulatory system overall). 18 June 2020.

Notes

(Edited to add text)