Science relies on replicability and reproducibility to differentiate valuable insights from chance occurrences, fraud, or misconduct. The cost of irreproducible research alone is estimated to be $28 billion annually. A recent PNAS paper proposed using machine learning (ML) to identify non-replicable findings, potentially aiding in the decision of which studies to manually replicate. However, our previous work has shown flaws in models used to predict social outcomes, and these flaws are often inherent and unfixable. Collaboration with experts revealed major drawbacks of the ML model discussed in the paper, including limited training data and susceptibility to gaming. Overall, ML for replication is unlikely to be useful for important decisions in science.
https://www.aisnakeoil.com/p/machine-learning-is-useful-for-many