"This "argument" by the "scientist" doesn't IMO ..."


by Eyal Roth Mar 27 2019

Scientist: So you're actually more of a computer science guy than an experimentalist yourself\. Why does this not surprise me? It's not impossible that some better statistical system than p\-values could exist, but I'd advise you to respect the wisdom of experience\. The fact that we know what p\-hacking is, and are currently fighting it, is because we've had time to see where the edges of the system have problems, and we're figuring out how to fight those problems\. This shiny new system will also have problems; you just have no idea what they'll be\. Perhaps they'll be worse\.

This "argument" by the "scientist" doesn't IMO represent how a true experimentalist would approach the issue; they would not necessarily be so opposed to trying new ways of improving their methods, as long as it is done step by step without replacing the entire system over night (just like the "bayesian" explains in the next paragraph).

This is also a bit side-tracking as it opens up the topic of how much more "experience" computer scientists have given the simpler and much more reproducible systems they're dealing with -- especially in the modern commercial world -- in contrast with natural sciences (I'm a programmer myself, so I'm a bit biased on this).