"If you dump enough computin..."


by Eliezer Yudkowsky Mar 28 2016 updated Mar 28 2016

If you dump enough computing power into hill-climbing optimization, within a Turing-general policy space, using a consequentialist criterion for fitness, it totally spits out a superintelligence not aligned with the original selection criterion. That's where humans come from in the first place.