Nick Bostrom on Environment - Dictionary of Arguments
Environment/simulation/superintelligence/decision-making/rewards/Bostrom: The AI might assign a substantial probability to its simulation hypothesis, the hypothesis that it is living in a computer simulation. Even today , many AIs inhabit simulated worlds - worlds consisting of geometric line drawings, texts, chess games, or simple virtual realities, and in which the laws of physics deviate sharply from the laws of physics that we believe govern the world of our own experience.
A mature superintelligence could create virtual worlds that appear to its inhabitants much the same as our world appears to us. The inhabitants would not necessarily be able to tell whether their world is simulated or not; (…).
How an AI would be affected by the simulation hypothesis depends on its values. >Values/superintelligence/Bostrom.
Simulation/reward/decision-making/superintelligence/Bostrom: The decision-making of an AI with goals that are easily resource-satiable may therefore - if it assigns a high probability to the simulation hypothesis - be dominated by considerations about which actions would produce the best result if its perceived world is a simulation.
Cooperation: if an AI with resource-satiable final goals believes that in most simulated worlds that match its observations it will be rewarded if it cooperates (but not if it attempts to escape its box or contravene the interests of its creator) then it may choose to cooperate. We could therefore find that even an AI with a decisive strategic advantage, one that could in fact realize its final goals to a greater extent by taking over the world than by refraining from doing so, would nevertheless balk at doing so. >Goals/superintelligence/Omohundro._____________Explanation of symbols: Roman numerals indicate the source, arabic numerals indicate the page number. The corresponding books are indicated on the right hand side. ((s)…): Comment by the sender of the contribution. Translations: Dictionary of Arguments The note [Author1]Vs[Author2] or [Author]Vs[term] is an addition from the Dictionary of Arguments. If a German edition is specified, the page numbers refer to this edition.
Superintelligence. Paths, Dangers, Strategies Oxford: Oxford University Press 2017