Coherence theorems

https://arbital.com/p/coherence_theorems

by Eliezer Yudkowsky Feb 7 2017

A 'coherence theorem' shows that something bad happens to an agent if its decisions can't be viewed as 'coherent' in some sense. E.g., an inconsistent preference ordering leads to going in circles.


[summary: In the context of decision theory, a coherence theorem shows that bad things happen to an agent whose behavior or beliefs can't be viewed as having property X. Conversely, an agent that doesn't stumble over its own feet in this sense, must be acting in a way that we can view as the agent being coherent in sense X.

Or as Steve Omohundro put it: If you prefer being in San Francisco to being in San Jose, prefer being in Oakland to being in San Francisco, and prefer being in San Jose to being in Oakland, you're going to waste a lot of money on taxi rides.

(In other words: If we can't view an agent's local choices as being coherent with some consistent global preference ordering, the agent is executing qualitatively dominated strategies. This is the sort of thing that coherence theorems say.)]

A tutorial introducing this concept exists here.

In the context of decision theory, "coherence theorems" are theorems saying that an agent's beliefs or behavior must be viewable as consistent in way X, or else penalty Y happens.

E.g., suppose we're talking about an agent's preference in pizza toppings. Let us say an agent locally prefers pizza topping A over pizza topping B if, offered a choice between a slice of A pizza and a slice of B pizza, the agent takes the A pizza.

Then suppose some agent:

Suppose also that at least the first preference is strong enough that, e.g. the agent would pay one penny to switch from pineapple pizza to onion pizza. %note: We can also say, e.g., that the agent would spend some small amount of time and effort to reach out and change pizza slices, even if this did not directly involve spending money.%

Then we can, e.g.:

Now the agent has the same pineapple pizza slice it started with, and is strictly one penny poorer. This is a qualitatively dominated strategy--the agent could have pursued a better strategy that would end with the same slice of pineapple pizza plus one penny. %note: Or without having lost whatever other opportunity costs, including the simple expenditure of time, it sacrificed to change pizza slices.%

Or as Steve Omohundro put it: If you prefer being in San Francisco to being in San Jose, prefer being in Oakland to being in San Francisco, and prefer being in San Jose to being in Oakland, you're going to waste a lot of money on taxi rides.

This in turn suggests that we might be able to prove some kind of theorem saying, "If we can't view the agent's behavior as being coherent with some consistent global preference ordering, the agent must be executing dominated strategies."

Broadly speaking, this is the sort of thing that coherence theorems say. Although nailing down caveats and generalizing to continuous spaces etcetera, often makes the standard proofs a lot more complicated than the above argument suggests.

Another class of coherence theorems says, "If some aspect of your decisions or beliefs has coherence properties X, we can map it onto mathematical structure Y." For example, other coherence theorems show that we can go from alternative representations of belief and credence, like log odds, to the standard form of probabilities, given assumptions like "If the agent sees piece of evidence A and then piece of evidence B, the agent's final belief state is the same as if it sees evidence B and then evidence A."

Coherence theorems generally point at consistent utility functions, consistent probability assignments, local decisions consistent with expected utility, or belief updates consistent with Bayes's Rule.

Since relaxing the assumptions used in a coherence theorem is an improvement on that theorem (and hence good for a publication), the total family of coherence theorems is rather large and very technical.

Coherence theorems are relevant because, e.g.:

Extremely incomplete list of some coherence theorems in decision theory

(somebody fill this out more, please)