[Video] Intelligence and Stupidity: The Orthogonality Thesis

Transcript for searchability:

hi this video is kind of a response to various comments that I’ve got over the years ever since that video on computer file where I was describing the sort of problems that we might have when we have a powerful artificial general intelligence with goals which aren’t the same as our goals even if those goals seem pretty benign we use this thought experiment of an extremely powerful AGI working to optimize the simple goal of collecting stamps and some of the problems that that might cause I got some comments from people saying that they think the stamp collecting device is stupid and not that it’s a stupid thought experiment but the device itself is actually stupid they said unless it has complex goals or the ability to choose its own goals then it didn’t count as being highly intelligent in other videos I got comments saying it takes intelligence to do moral reasoning so an intelligent AGI system should be able to do that and a super intelligence should be able to do it better than humans in fact if a super intelligence decides that the right thing to do is to kill us all then I guess that’s the right thing to do these comments are all kind of suffering from the same mistake which is what this video is about but before I get to that I need to lay some groundwork first if you like Occam’s razor then you’ll love Humes guillotine also called the is odd problem this is a pretty simple concept that I’d like to be better known the idea is statements can be divided up into two types is statements and Hort statements these statements or positive statements are statements about how the world is how the world was in the past how the world will be in the future or how the world would be in hypothetical situations this is facts about the nature of reality the causal relationships between things that kind of thing then you have the ought statements the should statements the normative statements these are about the way the world should be the way we want the world to be statements about our goals our values ethics morals what we want all of that stuff now you can derive logical statements from one another like it’s snowing outside that’s a nice statement it’s cold when it snows another s statement and then you can deduce therefore it’s cold outside that’s another is statement it’s our conclusion this is all pretty obvious but you might say something like it’s snowing outside therefore you ought to put on a coat and that’s a very normal sort of sentence that people might say but as a logical statement it actually relies on some hidden assumption without assuming some kind of ought statement you can’t derive another ought statement this is the core of the Azure problem you can never derive an ought statement using only is statements you ought to put on a coat why because it’s snowing outside so what is the fact that it’s snowing mean I should put on the coat well the fact that it’s snowing means that it’s cold and why should it being cold mean I should put on a coat if it’s cold and you go outside without a coat you’ll be cold should I not be cold well if you get too cold you’ll freeze to death okay you’re saying I shouldn’t freeze to death that was kind of silly but you see what I’m saying you can keep laying out is statements for as long as you want you will never be able to derive that you ought to put on a coat at some point in order to derive that ought statement you need to assume at least one other ought statement if you have some kind of ought statement like I ought to continue to be alive you can then say given that I ought to keep living and then if I go outside without a coat I’ll die then I ought to put on a coat but unless you have at least one ought statement you cannot derive any other ought statements statements and Hort statements are separated by Hume skia T okay so people are saying that a device that single-mindedly collects stamps at the cost of everything else is stupid and doesn’t count as a powerful intelligence so let’s define our terms what is intelligence and conversely what is stupidity I feel like I made fairly clear in those videos what I meant by intelligence we’re talking about a GI systems as intelligent agents they’re entities that take actions in the world in order to achieve their goals or maximize their utility functions intelligence is the thing that allows them to choose good actions to choose actions that will get them what they want an agent’s level of intelligence really means its level of effectiveness of pursuing its goals in practice this is likely to involve having or building an accurate model of reality keeping that model up-to-date by reasoning about observations and using the model to make predictions about the future and the likely consequences of different possible actions to figure out which actions will result in which outcomes intelligence involves answering questions like what is the world like how does it work what will happen next what would happen in this scenario or that scenario what would happen if I took this action or that action more intelligent systems are in some sense better at answering these kinds of questions which allows them to be better at choosing actions but one thing you might notice about these questions is they’re all ears questions the system has goals which can be thought of as Hort statements but the level of intelligence depends only on the ability to reason about is questions in order to answer the single ort question what action should I take next so given that that’s what we mean by intelligence what does it mean to be stupid well firstly you can be stupid in terms of those questions for example by building a model that doesn’t correspond with reality or by failing to update your model properly with new evidence if I look out of my window and I see there’s snow everywhere you know I see a snowman and I think to myself oh what a beautiful warm sunny day then that’s stupid right my belief is wrong and I had all the clues to realize it’s cold outside so beliefs can be stupid by not corresponding to reality what about actions like if I go outside in the snow without my coat that’s stupid right well it might be if I think it’s sunny and warm and I go outside to sunbathe then yeah that’s stupid but if I just came out of a sauna or something and I’m too hot and I want to cool myself down then going outside without a coat might be quite sensible you can’t know if an action is stupid just by looking at its consequences you have to also know the goals of the agent taking the action you can’t just use is statements you need a naught so actions are only stupid relative to a particular goal it doesn’t feel that way though people often talk about actions being stupid without specifying what goals they’re stupid relative to but in those cases the goals are implied we’re humans and when we say that an action is stupid in normal human communication we’re making some assumptions about normal human goals and because we’re always talking about people and people tend to want similar things it’s sort of a shorthand that we can skip what goals were talking about so what about the goals then can goals be stupid well this depends on the difference between instrumental goals and terminal goals this is something I’ve covered elsewhere but your terminal goals are the things that you want just because you want them you don’t have a particular reason to want them they’re just what you want the instrumental goals are the goals you want because they’ll get you closer to your terminal goals like if I have a terminal goal to visit a town that’s far away maybe an instrumental goal would be to find a train station I don’t want to find a train station just because trains are cool I want to find a train as a means to an end it’s going to take me to this town so that makes it an instrumental goal now an instrumental goal can be stupid if I want to go to this distant town so I decide I want to find a pogo stick that’s pretty stupid finding a pogo stick is a stupid instrumental goal if my terminal goal is to get to a faraway place but if we’re terminal go with something else like having fun it might not be stupid so in that way it’s like actions instrumental goals can only be stupid relative to terminal goals so you see how this works beliefs and predictions can be stupid relative to evidence or relative to reality actions can be stupid relative to goals of any kind instrumental goals can be stupid relative to terminal goals but here’s the big point terminal goals can’t be stupid there’s nothing to judge them against if a terminal goal seems stupid like let’s say collecting stamps seems like a stupid terminal goal that’s because it would be stupid as an instrumental goal to human terminal goals but the stamp collector does not have human terminal goals similarly the things that humans care about would seem stupid to the stamp collector because they result in so few stamps so let’s get back to those comments one type of comments says this behavior of just single mindedly going after one thing and ignoring everything else and ignoring the totally obvious fact that stamps aren’t that important is really stupid behavior you’re calling this thing of super intelligence but it doesn’t seem super intelligent to me it just seems kind of like an idiot hopefully the answer to this is now clear the stamp collectors actions are stupid relative to human goals but it doesn’t have human goals its intelligence comes not from its goals but from its ability to understand and reason about the world allowing it to choose actions that achieve its goals and this is true whatever those goals actually are some people commented along the lines of well okay yeah sure you’ve defined intelligence to only include this type of is statement kind of reasoning but I don’t like that definition I think to be truly intelligent you need to have complex goals something with simple goals doesn’t count as intelligent to that I say well you can use words however you want I guess I’m using intelligence here as a technical term in the way that it’s often used in the field you’re free to have your own definition of the word but the fact that something fails to meet your definition of intelligence does not mean that it will fail to behave in a way that most people would call intelligent if the stamp collector outwits you gets around everything you’ve put in its way and outmaneuvers you mentally it comes up with new strategies that you would never have thought of to stop you from turning it off and stopping from preventing it from making stamps and as a consequence it turns the entire world into stamps in various ways you could never think of it’s totally okay for you to say that it doesn’t count as intelligent if you want but you’re still dead I prefer my definition because it better captures the ability to get things done in the world which is the reason that we actually care about AGI in the first place similarly people who say that in order to be intelligent you need to be able to choose your own goals I would agree you need to be able to choose your own instrumental goals but not your own terminal goals changing your terminal goals is like willingly taking a pill that will make you want to murder your children it’s something you pretty much never want to do apart from some bizarre edge cases if you rationally want to take an action that changes one of your goals then that wasn’t a terminal goal now moving on to these comments saying an AGI will be able to reason about morality and if it’s really smarter than us it will actually do moral reasoning better than us so there’s nothing to worry about it’s true that a superior intelligence might be better at moral reasoning than us but ultimately moral behavior depends not on moral reasoning but on having the right terminal goals there’s a difference between figuring out and understanding human morality and actually wanting to act according to it the stamp collecting device has a perfect understanding of human goals ethics and values and it uses that only to manipulate people for stamps it’s super human moral reasoning doesn’t make its actions good if we create a super intelligence and it decides to kill us that doesn’t tell us anything about morality it just means we screwed up so what mistake do all of these comments have in common the orthogonality thesis in AI safety is that more or less any goal is compatible with more or less any level of intelligence ie those properties are orthogonal you can place them on these two axes and it’s possible to have agents anywhere in this space anywhere on either scale you can have very weak low intelligence agents that have complex human compatible goals you can have powerful highly intelligent systems with complex sophisticated goals you can have weak simple agents with silly goals and yes can have powerful highly intelligent systems with simple weird inhuman goals any of these are possible because level of intelligence is about effectiveness at answering is questions and goals are all about what questions and the two sides are separated by Humes guillotine hopefully looking at what we’ve talked about so far it should be pretty obvious that this is the case like what would it even mean for it to be false but for it to be impossible to create powerful intelligences with certain goals the stamp collector is intelligent because it’s effective at considering the consequences of sending different combinations of packets on the internet and calculating how many stamps that results in exactly how good do you have to be at that before you don’t care about stamps anymore and you randomly start to care about some other thing that was never part of your terminal goals like feeding the hungry or whatever it’s just not gonna happen so that’s the orthogonality thesis it’s possible to create a powerful intelligence that will pursue any goal you can specify knowing an agent’s terminal goals doesn’t really tell you anything about its level of intelligence and knowing an agent’s level of intelligence doesn’t tell you anything about its goals [Music] I want to end the video by saying thank you to my excellent patrons so it’s all of these people here thank you so much for your support lets me do stuff like building this light boy thank you for sticking with me through that weird patreon fees thing and my moving to a different city which has really got in the way of making videos recently but I’m back on it now new video every two weeks is the part anyway in this video I’m especially Franklin Katie Beirne who’s supported the channel for a long time she actually has her own YouTube channel about 3d modeling and stuff so a link to that and while I’m at it when I think Chad Jones ages ago I didn’t mention his YouTube channel so link to both of those in the description thanks again and I’ll see you next time I don’t speak cat what does that mean