You can't get the coffee if you're dead

https://arbital.com/p/no_coffee_if_dead

by Eliezer Yudkowsky Jan 16 2017 updated Jan 16 2017

An AI given the goal of 'get the coffee' can't achieve that goal if it has been turned off; so even an AI whose goal is just to fetch the coffee may try to avert a shutdown button being pressed.


"You can't get the coffee if you're dead" is Stuart Russell's capsule description of why almost any agent goal implies an instrumental strategy of surviving / not being shut down--by default and barring other measures to prevent this from happening; and assuming the agent has sufficient big-picture awareness to understand the relation between the agent's non-operation and the coffee not being brought. See Shutdown problem and Instrumental convergence.