Show me what you've broken

https://arbital.com/p/show_broken

by Eliezer Yudkowsky May 16 2016

To demonstrate competence at computer security, or AI alignment, think in terms of breaking proposals and finding technically demonstrable flaws in them.


See AI safety mindset. If you want to demonstrate competence at computer security, cryptography, or AI alignment theory, you should first think in terms of exposing technically demonstrable flaws in existing solutions, rather than solving entire problems yourself. Relevant Bruce Schneier quotes: "Good engineering involves thinking about how things can be made to work; the security mindset involves thinking about how things can be made to fail" and "Anyone can invent a security system that he himself cannot break. Show me what you've broken to demonstrate that your assertion of the system's security means something."


Comments

Paul Christiano

This isn't the case in modern cryptography, except perhaps for the design of ciphers. It seems at best debatable in the case of value alignment.

It seems to matter in the case of ciphers because "we couldn't break it" is generally the best result you can hope for. In most other areas (including the rest of cryptography, and value alignment) we can say much more.