printouts
{"Answer":["# ［https://www.youtube.com/watch?v꞊hEUO6pjwFOo The Orthogonality Thesis］: AI could have almost any goal while at the same time having high intelligence (aka ability to succeed at those goals). This means that we could build a very powerful agent which would not necessarily share human-friendly values. For example, the classic ［https://www.lesswrong.com/tag/paperclip-maximizer paperclip maximizer］ thought experiment explores this with an AI which has a goal of creating as many paperclips as possible, something that humans are (mostly) indifferent to, and as a side effect ends up destroying humanity to make room for more paperclip factories.\n# ［https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile Complexity of value］: What humans care about is not simple, and the space of all goals is large, so virtually all goals we could program into an AI would lead to worlds not valuable to humans if pursued by a sufficiently powerful agent. If we, for example, did not include our value of diversity of experience, we could end up with a world of endlessly looping simple pleasures, rather than beings living rich lives.\n# ［https://www.youtube.com/watch?v꞊ZeecOKBus3Q Instrumental Convergence］: For almost any goal an AI has there are shared ‘instrumental’ steps, such as acquiring resources, preserving itself, and preserving the contents of its goals. This means that a powerful AI with goals that were not explicitly human-friendly would predictably both take actions that lead to the end of humanity (e.g. using resources humans need to live to further its goals, such as replacing our crop fields with vast numbers of solar panels to power its growth, or using the carbon in our bodies to build things) and prevent us from turning it off or altering its goals."],"StampedBy":["plex"],"Tags":[{"fulltext":"Orthogonality thesis","fullurl":"https://stampy.ai/wiki/Orthogonality_thesis","namespace":0,"exists":"1","displaytitle":"orthogonality thesis"},{"fulltext":"Complexity of value","fullurl":"https://stampy.ai/wiki/Complexity_of_value","namespace":0,"exists":"1","displaytitle":"complexity of value"},{"fulltext":"Instrumental convergence","fullurl":"https://stampy.ai/wiki/Instrumental_convergence","namespace":0,"exists":"1","displaytitle":"instrumental convergence"}]}
answer
["# ［https://www.youtube.com/watch?v꞊hEUO6pjwFOo The Orthogonality Thesis］: AI could have almost any goal while at the same time having high intelligence (aka ability to succeed at those goals). This means that we could build a very powerful agent which would not necessarily share human-friendly values. For example, the classic ［https://www.lesswrong.com/tag/paperclip-maximizer paperclip maximizer］ thought experiment explores this with an AI which has a goal of creating as many paperclips as possible, something that humans are (mostly) indifferent to, and as a side effect ends up destroying humanity to make room for more paperclip factories.\n# ［https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile Complexity of value］: What humans care about is not simple, and the space of all goals is large, so virtually all goals we could program into an AI would lead to worlds not valuable to humans if pursued by a sufficiently powerful agent. If we, for example, did not include our value of diversity of experience, we could end up with a world of endlessly looping simple pleasures, rather than beings living rich lives.\n# ［https://www.youtube.com/watch?v꞊ZeecOKBus3Q Instrumental Convergence］: For almost any goal an AI has there are shared ‘instrumental’ steps, such as acquiring resources, preserving itself, and preserving the contents of its goals. This means that a powerful AI with goals that were not explicitly human-friendly would predictably both take actions that lead to the end of humanity (e.g. using resources humans need to live to further its goals, such as replacing our crop fields with vast numbers of solar panels to power its growth, or using the carbon in our bodies to build things) and prevent us from turning it off or altering its goals."]