"It's easy to equivocate bet..."


by Paul Christiano Jan 1 2016

It's easy to equivocate between "can be viewed as" and "is." Indeed, any rational agent "can be viewed as" an expected utility maximizer, but it need not have any internal architecture resembling such a maximizer. And in particular, the utility-function-being-maximized need not be represented explicitly.

Most of the actual oomph from decreeing something an expected utility maximizer seems to come from these additional assumptions, which aren't delivered by the relevant theorems. All the theorems give you is a characterization of the agent's attitude towards uncertainty (and so e.g. they have no content when there is no uncertainty).

(I expect the author doesn't often make this mistake, but it is pretty common in the broader LessWrong crowd.)