On inner and outer alignment, and their confusion

There have been multiple posts that have discussed inner and outer alignment and it seems like people are operating with slightly different definitions of these terms. This article aims to give a clear and general explanation of inner and outer alignment and clarify some causes of confusion and disagreement. Definitions I will start with what I consider to be a reasonable definition of the inner and outer alignment problems: Outer alignment problem: how do we design a test to determine whether or not an AI does what we want given a particular input? Other possible phrasings:

• How do we design a good utility function?

• How do we design a good way to rank possible worlds according to our values? Inner alignment problem: how do we ensure an AI always passes the test, without providing all possible inputs? Other possible phrasings:

• Given a good utility function, how do we ensure an AI actually optimises for this utility function?

• Given a good way to rank possible worlds according to our values, how do we ensure the AI optimises for the best possible world in all scenarios? In the limit of infinite data (and time, compute), inner alignment does not exist as we can test every possible input /​ output pair for the AI. Of course this is impossible in practice, but I find this fact a useful tool to help me decide whether something is an inner or outer alignment problem. I ask myself, if I could test with infinite data, would this problem still exist? Why must we solve inner and outer alignment? If we do not solve the inner alignment problem, the AI may behave in a way that passes the test given all inputs humans have provided to it, but does not pass the test consistently over all possible inputs. We can call this AI "deceptively aligned". A simple example of this is, say we provide inputs x, y and z, and our test is "does the AI output x + y + z?". Given 500 training examples, there is a whole family of functions that the AI could implement, besides x + y + z, that would still pass all 500 tests. For example, it could implement "return x + y + z if all inputs are in the range 0 to 1000 else return 2″. If we do not solve the outer alignment problem, we fail to filter out AI’s that don’t do what we want, in general. An example of this is, if our test is "does the AI’s outputted intervention reduce CO2 emissions", a solution that involves killing humans could pass. We have misspecified the test and so failed at outer alignment. Things that cause confusion There have been some debates about what counts and inner and outer alignment. One source of confusion is how do we define "the test" or "the objective"? Originally, I wrote that "does the AI output x + y + z?" is a possible test, and inner misalignment can occur if we only test with certain values of x, y and z. However, one can say that this is a misleading claim about what test we are actually using. I could rephrase the test I am actually using in this scenario as "does the AI output x + y + z given inputs drawn from a particular set?". Now, this test is obviously bad, because what we actually want is x + y + z for all inputs. And this looks like an outer alignment problem—the test itself is wrong; it is not a good proxy for what we actually want. In "Inner Alignment Failures" Which Are Actually Outer Alignment Failures, John Wentworth mentions that our use of birth control is actually an outer misalignment problem with respect to the optimisation process of evolution because "reproductive fitness in the ancestral environment" is not aligned with "reproductive fitness in the modern environment". This is a move which encapsulates some of the inputs to the test into the test itself (a bit like Currying in functional programming). This implies that at least to some extent, whether something is an inner or outer alignment problem is a property of the map, not the territory. Many inner alignment problems can be transformed into outer alignment problems by including some of the inputs of the test into the test specification itself, and then noticing that the test is bad. In the evolution case, this would be changing what we call the objective from "reproductive fitness" to "reproductive fitness in the ancestral environment". It may be more useful to think of the inner /​ outer alignment dichotomy as a way to split the alignment problem given a particular training scenario as opposed to a formal rule that can be used to put conceptual alignment problems into boxes. Another source of confusion is the use of optimisation and mesa-optimisation related terminology. The original Risks from Learned Optimization paper referred to AI’s possessing mesa-objectives. The mesa-objective is what the AI is actually optimising for, while the training objective is what we want it to optimise for. A reason why mesa-objectives as a concept could be useful for thinking about inner alignment is that we have some reason to believe that powerful AI’s will behave like optimisers. Optimisers optimise for some goal/​utility function, which in the case of the AI, we call the mesa-objective. The training process by which we obtain the AI itself is also an optimisation process, whose objective we refer to as the base objective. If we are able to detect a mesa-optimiser and show that its objective is not what we want it to be, then this is a clear way to show that the AI is inner-misaligned. However, using a definition that explicitly mentions mesa-objectives to define inner misalignment seems not general enough. Seeing an AI as an optimiser is more of a modelling choice, and in many cases it may not be easy to describe the AI’s behaviour in terms of an optimisation process with a well-defined objective function. In Risks from Learned Optimization: Introduction, Hubinger et. al. write "It might not be necessary to solve the inner alignment problem in order to produce safe, highly capable AI systems, as it might be possible to prevent mesa-optimizers from occurring in the first place.". However, given a more broad definition of inner misalignment, this is not true. Under a broader definition, given insufficient optimisation power and/​or training data, the solution found by the base optimiser may still be bad (even given a good base objective), and I suggest that we still refer to this as inner misalignment. Edit, based on discussion in comments and further research: This brings to light two possible ways of looking at inner misalignment:

• Inner misalignment is misalignment-that-is-not-outer-misalignment. Or inner misalignment is misalignment that occurs when the training objective is correct, but the training setup /​ training data itself causes a generalisation error. By this definition, "generalisation error" and "inner misalignment" are pointing to the same thing (although I would say that the former is describing the cause, while the latter is describing the effect)

• Inner misalignment is a form of generalisation error that occurs in cases when the AI itself is an optimiser (and so possesses a mesa-objective that can be misaligned with the base objective). This makes inner misalignment a subset of misalignment-that-is-not-outer-misalignment. Currently, my crux that would determine which of these definitions is most useful is whether or not deciding whether an AI is an optimiser, and finding its objective, is a well-defined procedure for powerful AIs.