Of course, the game is typically about costs and benefits. Saying "it is good to adopt the security mindset" is (often) an implicit claim about the relative costs of extra work vs. attacks. It's not totally clear if this article is making a similar claim, but it sounds like it.
In terms of costs and benefits, the AI case is quite different from typical security applications.
In the case of my disagreements with MIRI (which I think are relatively mild in this domain), here is how things look to me:
- There are a number of serious problems with existing approaches to AI control.
- Some of these failures seem very likely to be deal-breakers, for example "will not work unless AI development proceeds in a specific way that doesn't look particularly likely" or "will make AI systems 100x as expensive to run."
- Other failures are clearly troubling when looked at through the security mindset.
- I am mostly focused on correcting failures of the first kind. Researchers at MIRI are most interested in failures of the second kind.
In this case, the balance is not between "extra work" and "failure." It is between "failing because of X" and "failing because of Y." So to make the argument that addressing Y deserves priority, you need to do one of:
- Argue that, without further measures to address failure Y, we are doomed to fail. (+ some further claim about how dependent the solution to Y is on the solution to X, or something like that…)
- Argue that Y is a much more likely failure / tractable failure to fix / etc.
(The situation is a bit more subtle than that, since at this point we are mostly talking about arguments about whether a particular class of research problems is promising, or whether any AI control approaches informed by that research will inevitably be rejected by a more conservative approach. But that dichotomy gives the general picture.)
I don't think that such an argument has really been made yet, and the attempted arguments seem to mostly go through claims about future AI progress (especially with respect to fast takeoff) that I find pretty implausible.
So: my inclination is to go on being relatively unconservative (with respect to these particular concerns), and then to shift towards the security mindset once we start to understand the landscape of actually-possible-workable approaches to AI control.
My guess is that a similar strategy would have been appropriate in the early days of cryptography. The first order of business is to find the ideas needed for plausible practical infrastructure for secure communication.