When heavy optimization pressure on a system crystallizes it into an optimizer \- especially one that's powerful, or more powerful than the previous system, or misaligned with the previous system \- we could term the crystallized optimizer a "daemon" of the previous system\. Thus, under this terminology, humans would be daemons of natural selection\. If an AGI, after heavily optimizing some internal system, was suddenly taken over by an erupting daemon that cognitively wanted to maximize something that had previously correlated with the amount of available RAM, we would say this was a crystallized daemon of whatever kind of optimization that AGI was applying to its internal system\.
Eli's personal notes:
especially one that's powerful, or more powerful than the previous system
I'm quite interested in how often a crystalized optimizer will be more powerful / more intelligent than the system that gave rise to to it.
I have conflicting intuitions about how frequently that will be the case.