[summary: An agent is "reflectively stable" in some regard if it only self-modifies (or constructs successors) to think similarly in that regard. For example, an agent with a utility function that only values paperclips will construct successors that only value paperclips, so having a paperclip utility function is "reflectively stable" (and [goals_reflectively_stable so are most other utility functions]). Contrast "reflectively consistent".]
An agent is "reflectively stable" in some regard, if having a choice of how to construct a successor agent or modify its own code, the agent will only construct a successor that thinks similarly in that regard.
- In tiling agent theory, an [ expected utility satisficer] is reflectively consistent, since it will approve of building another EU satisficer, but an EU satisficer is not reflectively stable, since it may also approve of building an expected utility maximizer (it expects the consequences of building the maximizer to satisfice).
- Having a utility function that only weighs paperclips is "reflectively stable" because paperclip maximizers only try to build other paperclip maximizers.
If, thinking the way you currently do (in some regard), it seems unacceptable to not think that way (in that regard), then you are reflectively stable (in that regard).
[todo: untangle possible confusion about reflective stability not being "good" and wanting reflectively unstable agents because it seems bad to them if a paperclip maximizer stays a paperclip maximizer, or they imagine causal decision theorists building something incrementally saner than casual decision theorists.]