What role should evolutionary analogies play in understanding AI takeoff speeds?

https://www.lesswrong.com/posts/teD4xjwoeWc4LyRAD/what-role-should-evolutionary-analogies-play-in

Contents

What are the most compelling arguments for and against discontinuous versus continuous takeoffs? In particular, how should we think about the analogy from human evolution, and the scalability of intelligence with compute? In short, it seems that many people are quite confused about the use of evolution as an analogy in forecasting AI takeoff speeds (if not, then I certainly am!) and I hope to clarify things through this post. In particular, I’m focusing mostly on "how should we think about the analogy from human evolution", although I think this itself is a big enough question that I haven’t given a complete answer to it. I’ve made an effort to accurately represent existing arguments and stay up to date with new ones, but at the end of the day these are still my interpretations of them. Comments and feedback are very welcome, and all mistakes are my own.

Table of Contents

0 Overview

0.0 TL;DR

0.1 Introduction

Many arguments for AI takeoff speeds are based on analogies to evolution. Given that evolution has given rise to general intelligence, this seems like a reasonable thing to do. But how far can we extend this analogy? What are the important analogies that make this a valid comparison, and where does the analogy break down? This seems like an important question to answer, especially if many of our forecasts are based on evolutionary priors. In writing this post, I hope to get us a few steps closer to answering these questions. The first section focuses on clarifying the current state of the debate around AI takeoff speeds—the most common arguments, and my personal thoughts. The second section is a collection of what I think are the analogies and disanalogies between AI and evolution, in terms of takeoffs. These were compiled from posts and comments that I’ve seen on LessWrong, and some are based on my own impressions. I believe that our current understanding of these topics is insufficient, and that more work is needed, either to steelman existing arguments or find better ones (see 0.2 Motivation). The third section thus contains some suggestions for further research directions, with the caveat that I’m unsure about the importance of this area relative to other areas of AI safety. On the upside, it seems likely that these problems can be tackled by those without a technical background in AI.

0.2 Motivation

There are several reasons and motivations for why I decided to write this post. First, I’m worried that many evolution-based arguments about takeoff speeds might have flimsy premises, and I hope to increase discussion about this. I’ve also seen similar concerns echoed in several places. Second, the debate about takeoff speeds is spread over multiple blogs, papers, posts, and comments. One problem with the debate being so spread out is that different people are using different definitions for the same terms, and so I figured it might help to gather everything I could find in a single place. I think there are also other reasons to believe that work understanding the connections between AI and evolution can be useful. For one, it can help improve our understanding of related questions, e.g. "How are AI takeoff scenarios likely to arise? How should we prepare for them? How much should we trust evolutionary priors in forecasts about TAI timelines?" I also believe that we need stronger arguments before making political or research decisions. For instance, takeoffs that happen over multiple years could require different strategic considerations, and your views on this topic could influence which AI safety work you think is most important. Ben Garfinkel also pointed out that it can be risky to make large commitments on the basis of several blog posts, and I broadly agree with this.

0.3 Definitions

0.3.1 Takeoff of what?

If we’re looking at the question of AI takeoffs from a policy decision-making perspective, plausibly the main consideration should be the timelines for transformative AI (TAI). Holden Karnofsky defines this as AI that is "powerful enough to bring us into a new, qualitatively different future", and operationalises this in several ways for specificity. When talking about AI takeoffs, what I therefore mean is "the time period leading up to TAI" under this definition. I think this should be distinguished from artificial general intelligence (AGI), by which I mean an AI system that is able to exhibit intelligent behaviour in many domains. Contrast this to "narrow intelligence", which is only able to behave intelligently on a limited number of tasks, particularly in novel domains [1]. A lot of previous discussion about AI takeoffs has used AGI and TAI quite interchangeably—I think this is a reasonable position because general intelligence likely offers abilities like the ability "to tackle entirely new domains", thus making it plausible that an AGI would be transformative. However, I don’t want to rule out the possibility that TAI can arise with a fairly "narrow" AI, so in this post I’m most interested in the takeoff of TAI, which may or may not necessarily be AGI. For the sake of clarity, I’ll try to mention whether I’m referring to TAI generally or AGI specifically. Figure 1: Comparing continuous takeoff (left) and discontinuous takeoff (right). Here "progress" refers to some variable, say GDP, which we extrapolate to observe takeoff speed.### 0.3.2 Takeoff speeds A more disputed definition is how to measure takeoff speeds—this was discussed more extensively by Matthew Barnett in Distinguishing definitions of takeoff. The definition that I’ll use is based on the discontinuous progress investigation by AI Impacts, again based on decision-relevance.

0.3.3 Proxies

In practice, it can be quite hard to know what trajectory we should be paying attention to, and so operationalising the definitions of continuous and discontinuous takeoff can be challenging. One trajectory we could observe is the growth of world economic output, because it seems highly likely that a powerful AI would have large effects on economic productivity (especially if we’re comparing it to technological development during the industrial revolution). Even if we’re observing a quantitative measure of takeoff speed like economic output, it’s still hard to tell when exactly we’re satisfying the criteria for continuous takeoff—e.g. are we deviating from existing trends because of a discontinuity, or is this just random noise? One proxy that we can use to determine a discontinuity is one that Paul Christiano gives in Takeoff Speeds. Specifically, a continuous takeoff [4] is a "complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles." This captures our rough intuition as to what we should see from a discontinuity—that the growth is fast relative to what we might ordinarily expect, based on the overall time that the drivers of the growth have been present. While I think these definitions are broadly useful, I also believe that there are potentially some complications with these economic operationalisations of takeoff speeds. These are discussed a bit more in the section on analogies and disanalogies between AI and evolution.

0.4 Some assumptions about evolution

I mentioned my concerns about the "analogy to evolution" being rather vague. Part of this is due to the "analogy" being sloppy, but I suspect this is also due to "evolution" being a somewhat nebulous term. What exactly about evolution are we referring to? In the interests of concreteness, here are a few things about evolution that I’m assuming or relying upon for later parts of the post:

1 Evolutionary arguments for takeoff speeds

There have been quite a few arguments for takeoff speeds based on analogies to evolution. These arguments often follow the rough structure of:

1.1 Hominid variation

The classic argument for AI takeoffs is sometimes known as the "hominid variation" argument, an example of which is in Eliezer Yudkowsky’s post, Surprised by Brains. This argues that a continuous takeoff is more likely, by drawing analogies from the scaling of intelligence in humans. The basic argument is that evolution caused discontinuous progress in intelligence while it was (instrumentally) optimising for it, and AI is likely to do the same. A very similar argument is given in Intelligence Explosion Economics, where Yudkowsky argues that cognitive investment (e.g. greater brain compute, improved cognitive algorithms—not necessarily intelligence directly) leads to increased marginal returns in fitness. Here’s my attempt at breaking down this argument more carefully:

1.2 Changing selection pressures

One notable response to the hominid variation argument is given by Paul Christiano, in Arguments about fast takeoff. My current understanding of the argument is that it boils down to two main disagreements:

1.3 Cumulative cultural evolution

This argument is proposed by calebo in Musings on Cumulative Cultural Evolution and AI, and can be thought of as pushback against the "changing selection pressures" argument discussed earlier. One of the main counterarguments given by Christiano was the possibility that the relative evolutionary success of humans was not due to intelligence. The cumulative cultural evolution argument pushes back against this by referring to a conceptual model of human social learning. A highly simplified view of this says that the right mixture of social and asocial learning is necessary for fast development. Before that, we may get into an "overhang", e.g. where we have undergone a large amount of asocial learning, such that when we start doing social learning, development happens very quickly. The argument implies that we might need to consider a similar possibility in the development of TAI, and suggests an update towards a discontinuous takeoff.

1.3.1 My opinions on this debate

I think this argument raises quite a few additional questions, some of which I suspect already have answers but I’m unaware of:

1.4 Brains and compute

This argument from evolution is somewhat different from the hominid variation argument because it involves a quantitative comparison to what evolution was able to achieve with brains (albeit a very rough comparison). This works by asking the question, "how much was the brain able to do with X compute?", then using the answer to this to answer, "how much do we expect AI systems to be able to do with Y compute?". An example of this is given by Scott Alexander in Neurons and Intelligence: A Birdbrained Perspective, and another investigation by AI Impacts. If we look into our evolutionary history we observe a kind of "intelligence ladder". On the 80k podcast, Paul Christiano describes it this way (emphasis mine):

"A better way to compare [the abilities of AI relative to humans] is to look at what evolution was able to do with varying amounts of compute. If you look at what each order of magnitude buys you in nature, you’re going from insects to small fish to lizards to rats to crows to primates to humans. Each of those is one order of magnitude, roughly…" This argument thus gives a rough prior for what we might expect from the continued development of advanced AI systems. Neuron count [9,10]OrganismCapabilities0SpongeLimited sensory response Basic memory and understanding Logical reasoning, improved memory, etc. Self awareness Theory of mind, complex reasoning, etc. 231C. elegans~5 x 103Jellyfish104Leech2.5 x 105Ants106 Honeybee1.6 x 107Frog1.31 x 108Star-nosed mole3.10 x 108Pigeon2.25 x 109Dog9.6 x 109Brown bear2.8 x 1010Chimpanzee8.6 x 1010 (1015 connections [11])HumansTable 1: Using neuron count as a rough prior for the capabilities of agents, by comparing with organisms with different levels of intelligence. The capabilities column is very high in uncertainty and should be taken with a grain of salt. Scott Alexander also further states that we should expect abilities to "scale linearly-ish with available computing power", which is especially alarming given current compute trends, and if the scaling hypothesis is true. Looking at the intelligence ladder also suggests that we compare typical human performance with technologies relative to things found in biology or evolution. Moreover, similar arguments can be made by drawing comparisons with the energy efficiency of the brain, rather than their overall capabilities (e.g. because this is more measurable). I think there’s a tremendous amount of uncertainty in what exactly should go into the rightmost column of table 1, and that one could raise many questions about it. For instance, "to what extent are these capabilities due to intelligence, rather than other factors?" Another question might be whether the listed organisms might exhibit different behaviours in environments that differ from their natural habitat. This raises an issue of differences between observed behaviours and actual capabilities "out of distribution". For this reason I’ve mostly only listed very vague capabilities that I think are reasonably likely to generalise to different living environments. One could also claim that neuron counts are just the wrong approach altogether – for instance, Table 1 doesn’t include "African Elephant", with 2.5 x 1011 neurons below humans, but most would agree that humans are smarter than elephants. This raises the question of how much trust we should place in this analogy. These arguments don’t explicitly argue for whether takeoff is likely to be continuous or discontinuous; they instead provide a prior for what we should expect in the case of AI.

1.4.1 Side note: Dog-level AI

In the introduction, I mentioned that I didn’t want to rule out the possibility of TAI arising from sub-superintelligence, or perhaps even a fairly narrow intelligence. One way to frame this is to compare things with "human-level AI", for instance by talking about a "dog-level AI". But what does a "dog-level AI" actually mean? In the 2018⁄19 Overview of Technical AI Alignment episode on the AI Alignment Podcast, Rohin Shah and Buck Shlegeris define this to mean "a neural net that if put in a dog’s body replacing its brain would do about as well as a dog". They point out that such a system could plausibly become superhuman at many things in other environments—I think this suggests that the capabilities in table 1 are not upper bounds by any means, and potentially might say very little about how powerful "dog-level" systems actually are. So these capabilities really should be taken with a grain of salt!

1.5 Biological anchors

A close relative of the previous argument is to look at the total compute done over evolution and to use this as a prior. The best example of this is probably Ajeya Cotra’s Draft report on AI timelines, which uses biological features ("anchors") to develop a quantitative forecast of TAI timelines. This report wasn’t looking at AI takeoff speeds specifically, but I still think that this analogy is an important one. The forecast uses six biological anchors, and the one that we’re most interested in is the evolution anchor, describing the total compute over evolution from earliest animals to modern humans. With this anchor, we’re treating all of evolution as a single overarching algorithm, and asking how much compute it has taken to get to human-level intelligence [12]. Cotra gives a value of about 1041 floating point operations (FLOPs) for this, a number derived from Joseph Carlsmith’s report, How Much Computational Power Does It Take to Match the Human Brain?**. Cotra assigns 10% weight to the evolution anchor in the resulting forecast. Another intuition we might have is that evolution has done a significant amount of "pretraining" via neural architecture search, yielding the biological neural networks (i.e. our brains) that we’re familiar with today. Given that humans have hand-engineered artificial neural network architectures like RNNs, we might expect that the appropriate benchmark is the amount of compute required to "train" a person’s brain over the course of their lifetime. This describes the lifetime anchor, which predicts 1024 FLOPs to reach TAI. Cotra discusses this in much more detail in the report, including why this anchor seems quite unlikely to be correct. As of December 2021, the most computationally expensive systems require around 1024 FLOPs for the final training run (see the Parameter, Compute and Data Trends in Machine Learning project for more details on compute calculations). We can think of the evolution anchor as an upper bound for how much compute we would need to reach TAI. Given that evolution is myopic and inefficient (e.g. for yielding intelligence), and it was able to yield general intelligence despite this, then it seems extraordinarily likely that we’ll reach AGI with the same amount of compute as over all of evolution—especially if our algorithms are more efficient. If we accept the results of the report as being broadly accurate, then we get a very rough lower bound for how slow a takeoff can be before reaching TAI. I think that a better understanding of these anchors plausibly leads to better understanding of the ways in which we might build AGI. Of course, one could reasonably worry that the results are severe overestimates (or at least the upper bounds are so high that they aren’t helpful), see for instance Yudkowsky’s critique of the report.

1.6 One algorithm and Whole Brain Emulation

Another argument that one could make is that intelligence is generally described using a single simple algorithm. In such a scenario, stumbling across this algorithm could lead to a sharp discontinuity in progress. My impression is that the main sources of disagreement about this argument are:

1.7 Anthropic effects

One reason to be skeptical about the "one algorithm" argument is that observer-selection effects may conceal the difficulty in discovering an "algorithm for intelligence". Suppose there is a universe where finding this "intelligence algorithm" is very hard, and evolution was unsuccessful in discovering it. In this case, there would tautologically be no observer around to make the observation that "finding intelligence is very hard". At the same time, in a universe where finding the intelligence algorithm is very hard but evolution successfully stumbled upon it, the observers would conclude that "the intelligence algorithm can’t be that hard to find, because even blind evolution was able to happen upon it". The takeaway is that an observer sees intelligent life arise on their planet *regardless of how hard it is to find an algorithm for intelligence. * This argument is described in more detail in Shulman and Bostrom’s How Hard is Artificial Intelligence? Evolutionary Arguments and Selection Effects. Importantly, the authors also consider how different theories of observation-selection effects can be used to counter this objection to the "one algorithm" argument. This relies on (definitions from the LW wiki):

1.8 Drivers of intelligence explosions

The arguments in this section are in favour of discontinuous takeoff, and are mostly disanalogies to things that happen in evolution (more on this in section 2). In particular, they are of the form:

1.9 Summary

In this section I discussed a few of what I think are the most prominent arguments relating to takeoff speeds, that draw an analogy to evolution. I suspect that I’ve missed some arguments, but as far as I’m aware the ones listed have been the most prominent over the past few years.

2 Evaluating analogies and disanalogies

The first part of this section gives a big list of analogies and disanalogies, and the second considers plausible frameworks for thinking about them. I attempt to relate these analogies to the arguments from section 1, and also try to draw some preliminary conclusions. Some posts like Against evolution as an analogy for how humans will create AGI lay out reasons for why using evolution as an analogy can be problematic from the perspective of neuroscience. In this post I’ll be taking a different approach—I’m hoping to facilitate discussions that clarify existing arguments, rather than to push a particular point of view about whether or not evolution is a good analogy. By "analogy" I’m referring mostly to the definition of analogy in Daniel Kokotajlo’s post on "outside views". I don’t think any of the arguments from the previous section are rigorous enough to make them reference class forecasts. There are several different kinds of "analogies to evolution":

2.1 A non-exhaustive list of analogies and disanalogies

For each of these, I’ve characterised them as either an analogy or a disanalogy. I think it’s very likely that people will disagree with my classification, and that I’ll change my mind with more information. I’ve given each of these categorisations a certainty score that reflects my confidence in whether something is an analogy or a disanalogy (the higher the score, the more certain I am with the classification). I’d love to hear examples that support or contradict the claims here, or suggestions for further analogies/​disanalogies. The approach that I’ve taken is to go for breadth rather than depth—so I’ve also included some comparisons that I think are quite weak but are things that one "might want to consider", or analogies that people have made in the past. I hope that this will serve as a useful preliminary outline of the analogies and disanalogies that can be better formalised in further work.

Analogy: Timescales

Certainty score: 4⁄5 Example(s): Surprised by Brains According to this analogy, over evolutionary timescales, the development of human level intelligence was dramatically fast. If we compare this with the timescales of AI development, we should expect something similar (i.e. a discontinuous takeoff). I think you can split these analogies to timescales into two types:

Analogy: Necessary compute

Certainty score: 4⁄5 Example(s): Ajeya Cotra’s Draft Report on AI timelines, section on Brains and Compute (See 1.5 Biological anchors for more details.) This is a direct analogy to the amount of compute required for intelligence to arise in evolution. It informs possible quantitative priors that we can use in forecasts for AI takeoff scenarios, and are arguably most informative as providing upper bounds for when to expect TAI (and thus lower bounds for how slow a continuous takeoff can be). Possible weaknesses with this analogy include:

Analogy: Cumulative cultural evolution

Certainty score: 3⁄5 Example(s): Musings on Cumulative Cultural Evolution and AI (See 1.3 Cumulative cultural evolution for more details.) This is an analogy used to argue for discontinuous takeoff via a model of the discontinuous growth mechanisms in evolution, in turn implying the plausibility of similar mechanisms in TAI development. We could arguably also claim that there have been some small discontinuities in certain domains like Go. Possible weaknesses with this analogy include:

Analogy: Intelligence is non-hard

Certainty score: 4⁄5 Example(s): Hard Takeoff (See 1.6 One algorithm and Whole Brain Emulation for more details) This analogy claims that because even blind evolution was able to discover an algorithm for general intelligence, finding the right algorithm for general intelligence in TAI is boundedly difficult. While it’s not clear how easy or hard intelligence really is, although it plausibly does seem reasonably non-hard. Possible weaknesses with this analogy:

"...observed evolutionary history—the discontinuity between humans, and chimps who share 95% of our DNA—lightly suggests a critical threshold built into the capabilities that we think of as ‘general intelligence’, a machine that becomes far more powerful once the last gear is added." In evolution, thresholds can arise when a small change in one variable past a certain threshold value leads to a large change in another variable. Examples of these include threshold traits that explain why guinea pigs develop different numbers of digits. But these generally only describe small scale changes, rather than very large changes in traits. Possible weaknesses with this analogy:

Analogy: Number of neurons

Certainty score: 4⁄5 Example(s): 80k podcast with Paul Christiano A rough gauge we can use for the capabilities of AI systems of different sizes is by looking at the capabilities of organisms with brains of differing numbers of neurons (see 1.4 Brains and compute). Note that this doesn’t refer to the number of connections per neuron—in chapter 1 of Deep Learning, Goodfellow et al. point out that artificial neural networks have had close to the same number of connections per neuron as in the human brain for many years, but we’re still orders of magnitude away from having the same number of neurons. A related suggestion is to look at the ratio of content to architecture size, rather than just the architecture size—see Source code size vs learned model size in ML and in humans? Possible weaknesses:

Analogy: Structure of the learning system

Certainty score: 2⁄5 Example(s): My computational framework for the brain Evolution designed human brains with a particular structure, and perhaps there are important features of this that need to be mimicked when developing AI in order for the resulting systems to be generally intelligent. A natural comparison that we might make is based on the architectures and modularity of learning systems. Different parts of the brain have different functions, and modern ML systems often consist of several models designed for different purposes. As a simple example, a CNN has a head that learns a feature map, which is fed as an input vector into a fully connected neural net for classification (Richard Ngo calls this type of modularity architectural modularity, and distinguishes this from emergent modularity arising in a neural net from training). Some people argue that this modular structure makes it easier to evolve—parts can be added or removed from the system without affecting the function of other modules, although others contest this claim. Other related factors one might want to think about are:

Disanalogy: Training environments

Certainty score: 2⁄5 Example(s): AGI safety from first principles: Superintelligence In natural selection, populations become adapted to existing environments, not to future ones. This increases fitness relative to the context within which selection is happening. Perhaps AI training environments are going to be more conducive to a fast takeoff, because these are hand-designed by AI researchers rather than purely based on the current state of the natural environment. Some might counter that training of an AI in a virtual environment is insufficient; general intelligence can only arise in an environment that resembles the real biological one. However, this is potentially a pessimistic view, especially given that many features of biological environments can be simulated. Richard Ngo previously also discussed some ways in which different environments could bottleneck AGI development. Possible weaknesses:

Disanalogy: Order of training

Certainty score: 1⁄5 Example(s): Analogies and General Priors on Intelligence In Analogies and General Priors on Intelligence*,* riceissa and Sammy Martin point out the order of learned tasks is different for AI, as compared with human evolution:

"Language, for instance, was evolved very late along the human lineage, while AI systems have been trained to deal with language from much earlier in their relative development. It is unknown how differences such as this would affect the difficulty-landscape of developing intelligence." This is perhaps related to the analogy mentioned by Stuart Russell in Human Compatible*.* Specifically, Russell argues that there are certain milestones, like the ability to plan and understand language, before we can reach AGI. Possible disanalogies here include "how fast we’ll be able to reach all of these milestones" and "whether there is a fixed order in which the milestones are reached". A proxy for how fast AI takeoff occurs (if it does) would thus be something along the lines of, "how fast do you think we’ll be able to reach all of these milestones"? Possible weaknesses:

Disanalogy: Optimisation objectives

Certainty score: 3⁄5 Example(s): AGI safety from first principles: Superintelligence, Takeoff speeds Current AI systems tend to be trained to maximise performance on specific tasks (e.g. image classification), leading to relatively narrow capabilities. On the other hand, organisms in evolution need to deal with multiple different tasks (e.g. reasoning, image recognition, planning), and thus need to be more well-rounded to survive. But how exactly should we think of the "optimisation objectives" for AI systems? If we’re thinking about a single task like classification using a CNN, then the objective might be to "minimise the loss function". If we think more broadly, then one could also argue that the objectives for AI systems are to be useful to humans (by design, if we assume that the objective functions have been specified correctly). On the other hand, evolution is optimising for fitness. Possible weaknesses:

Disanalogy: Changing selection pressures

Certainty score: 4⁄5 Example(s): Takeoff speeds Paul Christiano argues that selection pressures won’t change in AI (since AI systems are being designed for usefulness to humans), but have constantly been changing throughout evolutionary history, and so a discontinuous takeoff in AI is less likely. By selection pressures, I’m referring to a driver of natural selection that favours particular traits, like "high social intelligence". One potential cause for these are environmental effects—fitness is defined relatively to the current environment, and so changes in the environment could change what natural selection (or a more general search algorithm) preferentially selects for. This could be induced by agents that modify the environment they are in; depending on how much control agents have in modifying the environment, they may be able to control the selection pressures in the way that they "desire" (similar to how humans build houses for shelter). Possible weaknesses:

Disanalogy: Data constraints

Certainty score: 2⁄5 Example(s): the scaling "inconsistency": openAI’s new insight (and Scaling Laws for Autoregressive Generative Modeling) This disanalogy is between how much data is available for "training" in evolution as opposed to AI. Currently, ML systems are partially bottlenecked by a lack of labelled data, rather than data in general, which I’ll term as a lack of "useful data". A corollary of this is that we might expect language models to eventually become data constrained. In evolution on the other hand, there is no shortage of useful data to train on—organisms gather information by interacting with the environment. Possible weaknesses:

Disanalogy: Energy costs

Certainty score: 3⁄5 Example(s): Training a single AI model can emit as much carbon as five cars in their lifetimes The brain seems to be very energy efficient, and machine learning models take a ton of energy consumption to train [16]. For instance, our brains are able to run on about 20W, which is comparable with the power of my desk lamp. In contrast, training of neural networks has led to increasing concerns about the carbon footprint of ML. Perhaps a feature to pay attention to is how the energy costs and efficiency of AI systems changes over time. This could be important because of tradeoffs that exist in the evolutionary environment. On the one hand, having a larger brain generally yields cognitive benefits that increase odds of survival and maximising fitness. On the other hand, having larger brains requires more energy costs, reducing energy that can be devoted toward reproduction and development, thus reducing fitness [17]. Possible weaknesses:

Disanalogy: Drivers of intelligence explosions

Uncertainty score: 5⁄5 Example(s): AGI Safety from first principles (See 1.8 Drivers of intelligence explosions for more details.) This class of disanalogies argues that there are important ways in which AI development is different from evolutionary growth, and that this changes the picture completely. Possible weaknesses:

2.2 Discussion

Based on the above, it seems like there are many different considerations that might plausibly need to be considered. In order for analogies to evolution to be useful in helping us draw conclusions, I think it would help to understand which properties need to be analogous and why. I think now is a good time to revisit the original question that inspired this post: "How should we think about the analogy to evolution?"

2.2.1 A possible framework: levels of analysis

The list of analogies and disanalogies is quite messy as it stands, and overall I’m very unsure about how to orient towards them. One possibility is to follow an approach that is a modified version of Marr’s Levels of Analysis (credits to Vael for suggesting this). According to Kraft and Griffiths:

"Marr (1982) famously argued that any information processing system can be analyzed at three levels, that of (1) the computational problem the system is solving; (2) the algorithm the system uses to solve that problem; and (3) how that algorithm is implemented in the "physical hardware" of the system." An example of this that I really like is proposed by Steven Byrnes, that separates the development of general intelligence into an "outer algorithm" and an "inner algorithm". Evolution****AIOuter algorithm (runs an automated search for the inner algorithm)Evolution searching over the space of possible cognitive architectures and algorithmsHumans searching over the space of AI system architectures and algorithmsInner algorithm (the general intelligence)The human brain algorithmE.g. SGDTable 3: Steven Byrnes’ description of the analogy to evolution involves breaking down the process of AGI development into two parts—an inner algorithm (for the trained intelligent agent) and an outer algorithm which searches for the inner algorithm. I think this way of viewing the list of analogies and disanalogies leads to a decent classification of the internal algorithmic aspects, but some analogies are also about the effects and capabilities of them. To include these other analogies, I suggest one other category, the "effects", yielding three levels:

2.2.2 Should we use evolution as an analogy?

So, should we even be using these analogies and disanalogies? We should probably be more confident in the conclusions derived from these comparisons if they are used as plausibility arguments, as opposed to purely quantitative measures, simply because a plausibility claim is weaker. This of course depends on how the quantitative measure is being done, and what it is being used for (e.g. I believe bioanchors can still be a useful approach as a rough proxy or for getting an upper bound for when we might reach TAI). I believe a lot of the value of these analogies could come from their information value (e.g. in helping us understand the mechanisms that could lead to different takeoff scenarios). As an example, understanding the role of modular systems in evolution could end up being very useful if it is indeed a large driver in the evolution of human intelligence. Also, if we’re looking at intelligence specifically, what other options do we have? I think we shouldn’t be too quick to dismiss analogies to evolution, depending on the use case. Overall, I think that the way in which analogies and disanalogies to evolution are being used is pretty reasonable. Personally, I think that using evolution as an analogy makes sense and is a good idea, as long as it remains exactly that—an analogy. My impression is that we would need a lot more evidence or understanding to be able to use evolution as evidence in a more rigorous fashion.

2.2.3 Relation to other arguments about takeoff speeds

Perhaps you’re thinking that most of these analogies just aren’t going to be very significant compared to other arguments about takeoffs in general, like the scaling hypothesis or concerns about an AI overhang. Given the uncertainties regarding evolution and the many possible disanalogies, I think existing arguments that include both evolutionary and other considerations make a lot of sense. But how much exactly? I don’t have a strong opinion on this, but I think a large source of variance here is how convincing you think the disanalogies are (especially the drivers of intelligence of explosions). Overall I’m unsure about more precise proportions as to how strongly to weigh evolutionary arguments against other ones (e.g. economic arguments) in forecasts.

2.3 Summary

The analogies and disanalogies that I mentioned previously are nowhere near exhaustive. Looking at things overall, I think it’s potentially useful or impactful to investigate which of these analogies or disanalogies are the most important and why.

3 Further work

In this section, I list some brief thoughts on what might potentially be useful questions to answer. I don’t have a good grasp of how tractable these are, or how likely it is that they’ll be highly useful for furthering understanding. I leave these considerations as an exercise for the reader.

3.1 Clarifying definitions

3.1.1 Measures of growth

This is an example of what I think is an ambiguity that often arises when talking about takeoffs, that I’m hoping to get some clarity on. Specifically, I think the choice of measure is quite important when extrapolating previous progress to determine whether a takeoff is continuous or not. One concern with the definition that I used previously, global economic output (see 0.3 Definitions), is the possibility that global economic output grows continuously, while another measure (like intelligence) grows discontinuously. Consider the example of LED efficiency: Figure 4: [Image source] Graphs showing the change in light source efficiency over time. The graph on the right shows the same graph as on the left, but with two rough curves highlighted—a continuous curve that represents continuous growth in lighting efficiency, and a discontinuous curve that represents growth in LED efficiency since first invention. We can observe several things:

3.1.2 Miscellaneous

Another confusion that I had when working on this project is how we came to the definitions of "continuous" and "discontinuous" takeoff in the first place. However, since I’m viewing things with an eye towards political decision-making, I’m under the impression that the continuous/​discontinuous distinction might not make as large of a difference as I’d thought (see 0.2 Motivation). More discussion of this can be found at Soft takeoff can still lead to decisive strategic advantage, and the subsequent review. I’m interested in better understanding exactly how slow the takeoff would need to be to make a big difference in our ability to respond to increasingly rapid AI development, and having terms that distinguish between these scenarios. In short, currently we’ve been asking, "if takeoff happens at a rate X, then will society be able to do Y in time?" I want to ask, "if we want to be able to do Y in time, what X is necessary?".

3.2 Clarifying understanding

If we temporarily ignore arguments about evolution altogether, my impression is that most people seem to think that a continuous takeoff is more likely (under Paul Christiano’s definition of "slow takeoff"). I’m unsure about why exactly this is the case, and I’m curious to hear what people think about this—is this because the evolutionary arguments in favour of a discontinuous takeoff, like the hominid variation argument, seem largely unconvincing? In general, I’d be curious to get a better idea of what the most relevant takeoff measures are, and the main drivers of takeoff speeds (as well as the role of evolutionary analogies in that). In particular, do you think that the role of evolutionary analogies are relatively small compared to other arguments about takeoff speeds? If so, why? I also think it would be good to look into specific cases of these analogies or disanalogies and steelmanning or critiquing them. Some topics in evolution that I think it could be valuable to look into include:

4 Conclusions

I wrote this post to raise and address some uncertainties around using evolution as an analogy for the development of TAI, and potentially encourage new work in this area that benefits political decision-making. In particular, I focus on arguments about the takeoff speeds of TAI, although the conclusions may apply more generally. The post split into three main sections:

Reviewing key arguments

A non-exhaustive list of analogies and disanalogies

Further work

Other things (in addition to the role of analogies/​disanalogies to evolution) that I think remain unclear include: