by Eliezer Yudkowsky Jul 16 2015 updated Dec 17 2015

What is "instrumental" in the context of Value Alignment Theory?

An 'instrumental strategy', 'instrumental goal', or 'subgoal' is an event E that an agent tries to bring about in order to bring about some other goal G. If you want to drink milk, then you need to drive to the store; in order to drive to the store, you need to be inside your car; in order to be inside your car, you need to open your car door. Thus 'be inside my car' and 'open my car door' are instrumental goals or instrumental strategies.

In conventional philosophy, an event is said to have "instrumental value" if it is useful for accomplishing some implied other set of goals, as distinguished from "terminal value" which is unconditional on future events. Since in VAT we have reserved the word 'value', we can't use that terminology here.

[todo: - Use a Bayes net formalism to explain why some formulations of expected utility make the key idea more or less transparent.]