Replicating replicability modeling of psychology papers

Aske Mottelson, Dimosthenis Kontogiorgos

Publikation: Artikel i tidsskrift og konference artikel i tidsskriftTidsskriftartikelForskningpeer review

Abstract

Youyou et al. (1) estimated the replicability of more than 14,000 psychology papers using a machine learning model, trained on main texts of 388 replicated studies. The authors identified mean replicability scores of psychological subfields. They also verified the causality of the model predictions; correlations between model predictions and study details the model was not trained on (i.e., P value and sample size) were reported.

In attempting replication, we identified important shortcomings of the approach and findings. First, the training data contain duplicated paper entries. Second, our analysis shows that the model predictions also correlate with variables that are not causal to replicability (e.g., language style). These issues impede the validity of the model output and thereby paint an erroneous picture of replication rates of the psychological science. In this letter, we attempt to mitigate these issues and nuance the findings of the original paper.
OriginalsprogEngelsk
TidsskriftProceedings of the National Academy of Sciences of the United States of America
Vol/bind120
Udgave nummer33
ISSN0027-8424
DOI
StatusUdgivet - 7 aug. 2023

Emneord

  • Replicability
  • Machine learning
  • Psychology
  • Causal inference
  • Model validation

Fingeraftryk

Dyk ned i forskningsemnerne om 'Replicating replicability modeling of psychology papers'. Sammen danner de et unikt fingeraftryk.

Citationsformater