"Regarding corporations: I ..."

Regarding corporations:

I have seen very few arguments about superintelligence that rest on epistemic efficiency. Roughly speaking, epistemic efficiency with respect to X might be interpreted as "smarter in every way than X." But we usually talk about systems that are "smarter in some ways than humans." And the safety problem doesn't seem to change in a qualitative way at the threshold of "smarter in every way." Nor does economic value, or most indicators of interest. The only measure on which that's a fundamental threshold is "how hard it is for humans to do anything useful" (but this is not an indicator people are talking about if they talk about corporations, for obvious reasons…)

So while I might agree that a corporation is not a superintelligence on Nick's definition, this doesn't seem to have much bearing on the way in which the analogy is invoked in discussions. In general, this notion of superintelligence is a sufficient condition for lots of interesting phenomena, but not a necessary condition for almost anything.

It just seems like you semantically disagreeing about the use of the word "superhuman." This seems like a missed opportunity to help communicate about the value alignment problem. (Also, though it's not precisely related, I think that people really do think about AI better when they think about it as "idealized corporation" rather than "idealized human." Corporatization seems to be a better baseline than anthropomorphization, though neither is great.)

To make things more concrete, I think it is roughly as reasonable to ask about the "value alignment problem for organizations," and that many solutions to the value alignment problem for AI will also be applicable to organizations (and conversely, if someone came to me with an actually good proposal for value alignment for organizations, I would consider it worth-looking-at). Of course I think that value alignment problem for AI systems is much more important, and so where the two problems are disanalogous I care about the AI version (and also I want to reserve undecorated "value alignment" as a technical term referring to the AI version of the problem). But that judgment is unrelated to the epistemic efficiency of corporations---I'd think the same thing even if corporations were epistemically efficient, and I presume so would you.

Your basic complaint with people's use of corporations as an analogy really seems to be that AI systems will become very much more powerful, and that they will never have the same peculiar mix of abilities as human organizations.

(Indeed, assuming that an agent is smarter in every way simply makes the safety problem easier, and many of our disagreements about safety are based on me being willing to assume something like "epistemic efficiency with respect to an average college graduate," at least as a first step.)