"> >This is like one step of..."

https://arbital.com/p/1j2

by Paul Christiano Jan 1 2016


This is like one step of ten in the act-based approach, and so to the extent that we disagree it seems important to clear that up.

I'm sorry if I seem troublesome or obstinate here.

I'm just pointing this out to clarify why I care about what may seem like a minor point (if you could make a safe genie, then there is a relatively clear research path to removing the human involvement). I don't care much about this point on its own, I'm mostly interested because this is one key step of the research project I'm outlining.

I don't have objections to going through basic points in great detail (either here or in the last discussion).

My possibly wrong or strawmanning instinctive model of one of our core disagreements is that, in general, Eliezer thinks "The problem of making a smarter-than-human intelligence that doesn't kill you, on the first try, is at least as metaphorically difficult as building a space shuttle in a realm where having the wrong temperature on one O-Ring will cause the massive forces cascading through the system to blow up and kill you, unless you have some clever meta-system that prevents that, and then the meta-system has to not blow up and kill you" and Paul does not feel quite the same sense of "if you tolerate enough minor-seeming structural problems it adds up to automatic death".

I agree that we have some disagreement about P(doom), though I assume that isn't fundamental (and is instead a function of disagreements about humanity's competence, the likely speed of takeoff, the character of the AI safety problem).

But I think that most of the practical disagreements we have, about where to focus attention or what research problems to work on, are more likely to be driven by different approaches to research rather than different levels of optimism.