You can't get the coffee if you're dead

by Eliezer Yudkowsky Jan 16 2017 updated Jan 16 2017

An AI given the goal of 'get the coffee' can't achieve that goal if it has been turned off; so even an AI whose goal is just to fetch the coffee may try to avert a shutdown button being pressed.

"You can't get the coffee if you're dead" is Stuart Russell's capsule description of why almost any agent goal implies an instrumental strategy of surviving / not being shut down--by default and barring other measures to prevent this from happening; and assuming the agent has sufficient big-picture awareness to understand the relation between the agent's non-operation and the coffee not being brought. See Shutdown problem and Instrumental convergence.