Meta-utility function

https://arbital.com/p/meta_utility

by Eliezer Yudkowsky Feb 13 2017

Preference frameworks built out of simple utility functions, but where, e.g., the 'correct' utility function for a possible world depends on whether a button is pressed.


A "meta-utility function" is a preference framework built by composing multiple simple utility functions into a more complicated structure. The point might be, e.g., to describe an [-agent] that optimizes different utility functions depending on whether a switch is pressed, or an agent that learns a 'correct' utility function by observing data informative about some Ideal target. For central examples see Utility indifference and Moral uncertainty.