"> Even so, while the output..."


by Paul Christiano Apr 30 2016

Even so, while the outputs are still abstract and not-yet-computed, Alice doesn't have much of a place to stand on which to appeal to Carol, Dennis, and Evelyn by saying, "But as a matter of morality and justice, you should have the AI implement my extrapolated volition, not Bob's!"

They may not have a moral argument, but they can surely have an argument.

And so on, this is a tiny fraction of the plausible alternatives. I don't really think that any is a strong Schelling point, and certainly none is so strong that you can't argue for one of the others.

You say that the purpose of not being a jerk is so that people can cooperate, rather than turning the development of AI into a conflict. If that's your goal, wouldn't the default approach be to give each individual enough influence to ensure that they have no incentive to defect? If you try to assign weight democratically, you are massively reducing the influence of many particular individuals, including almost every researcher, investor, and regulator. That does not seem like the most natural recipe for eliminating conflict!

As another way of putting it, suppose that I was to be made dictator of the world tomorrow. What should I do, if I wanted to not be a jerk? One proposal is to redistribute all resources equally amongst living humans. Another is to do nothing. People will justifiably object to both, I don't think there is a simple story about which is right (setting aside pragmatic concerns about feasibility).

You can try to get out of this, by claiming that the pie is going to grow so much that this kind of conflict is a non-issue. I think that's true to the extent that people just want to live happy, normal lives. But many people have preferences over what happens in the world, not only about their own lives. From an aggregative altruistic perspective these are the preferences that are really important, and they are almost necessarily in tension since realizing any of them demands some resources.


Eliezer Yudkowsky

I doubt it will satisfy you, but see the added "Selfish bastards" and "Why include everyone" sections.