The Importance of Heterogeneity in Large-Scale Replications

Matthew H. Goldberg, Sander van der Linden

Abstract


In a large-scale replication effort, Klein et al. (2018, https://doi.org/10.1177/2515245918810225) investigate the variation in replicability and effect size across many different samples and settings. The authors concluded that, for any given effect being studied, heterogeneity across samples and settings does not explain failures to replicate. In the current commentary, we argue that the heterogeneity observed indeed has implications for replication failures, as well as for statistical power and theory development. We argue that psychological scientific research questions should be contextualized—considering how historical, political, or cultural circumstances might affect study results. We discuss how a perspectivist approach to psychological science is a fruitful way for designing research that aims to explain effect size heterogeneity.

Keywords


heterogeneity; Many Labs 2; replication; perspectivism; context

Full Text: PDF HTML
Downloads: 707

https://doi.org/10.5964/jspp.v8i1.1187

Array ( [0] => Array ( [FORWARD_LINK] => ) [1] => Array ( [JOURNAL_CITE] => [ISSN] => 22146296 [JOURNAL_TITLE] => Energy Research & Social Science [JOURNAL_ABBREVIATION] => Energy Research & Social Science [ARTICLE_TITLE] => Validity of energy social research during and after COVID-19: challenges, considerations, and responses [CONTRIBUTORS] => [CONTRIBUTOR] => [GIVEN_NAME] => Array ( [0] => Michael J. Fell [1] => Laura Pagel [2] => Chien-fei Chen [3] => Matthew H. Goldberg [4] => Mario Herberz [5] => Gesche M. Huebner [6] => Siddharth Sareen [7] => Ulf J.J. Hahnel ) [VOLUME] => 68 [FIRST_PAGE] => 101646 [ITEM_NUMBER] => 101646 [YEAR] => 2020 [PUBLICATION_TYPE] => full_text [DOI] => 10.1016/j.erss.2020.101646 [FORWARD_LINK] => [BODY] => ) [2] => Array ( [JOURNAL_CITE] => [ISSN] => 2397-3374 [JOURNAL_TITLE] => Nature Human Behaviour [JOURNAL_ABBREVIATION] => Nat Hum Behav [ARTICLE_TITLE] => Replicating patterns of prospect theory for decision under risk [CONTRIBUTORS] => [CONTRIBUTOR] => [GIVEN_NAME] => Array ( [8] => Kai Ruggeri [9] => Sonia Alí [10] => Mari Louise Berge [11] => Giulia Bertoldo [12] => Ludvig D. Bjørndal [13] => Anna Cortijos-Bernabeu [14] => Clair Davison [15] => Emir Demić [16] => Celia Esteban-Serna [17] => Maja Friedemann [18] => Shannon P. Gibson [19] => Hannes Jarke [20] => Ralitsa Karakasheva [21] => Peggah R. Khorrami [22] => Jakob Kveder [23] => Thomas Lind Andersen [24] => Ingvild S. Lofthus [25] => Lucy McGill [26] => Ana E. Nieto [27] => Jacobo Pérez [28] => Sahana K. Quail [29] => Charlotte Rutherford [30] => Felice L. Tavera [31] => Nastja Tomat [32] => Chiara Van Reyn [33] => Bojana Većkalov [34] => Keying Wang [35] => Aleksandra Yosifova [36] => Francesca Papa [37] => Enrico Rubaltelli [38] => Sander van der Linden [39] => Tomas Folke ) [VOLUME] => 4 [ISSUE] => 6 [FIRST_PAGE] => 622 [YEAR] => 2020 [PUBLICATION_TYPE] => full_text [DOI] => 10.1038/s41562-020-0886-x [FORWARD_LINK] => [BODY] => [QUERY_RESULT] => [CROSSREF_RESULT] => ) )

Citations:

  • Michael J. Fell, Laura Pagel, Chien-fei Chen, Matthew H. Goldberg, Mario Herberz, Gesche M. Huebner, Siddharth Sareen, Ulf J.J. Hahnel (2020)
    Validity of energy social research during and after COVID-19: challenges, considerations, and responses
    Energy Research & Social Science, 68
    https://doi.org/10.1016/j.erss.2020.101646
  • Kai Ruggeri, Sonia Alí, Mari Louise Berge, Giulia Bertoldo, Ludvig D. Bjørndal, Anna Cortijos-Bernabeu, Clair Davison, Emir Demić, Celia Esteban-Serna, Maja Friedemann, Shannon P. Gibson, Hannes Jarke, Ralitsa Karakasheva, Peggah R. Khorrami, Jakob Kveder, Thomas Lind Andersen, Ingvild S. Lofthus, Lucy McGill, Ana E. Nieto, Jacobo Pérez, Sahana K. Quail, Charlotte Rutherford, Felice L. Tavera, Nastja Tomat, Chiara Van Reyn, Bojana Većkalov, Keying Wang, Aleksandra Yosifova, Francesca Papa, Enrico Rubaltelli, Sander van der Linden, Tomas Folke (2020)
    Replicating patterns of prospect theory for decision under risk
    Nature Human Behaviour, 4(6), p. 622(ff.)
    https://doi.org/10.1038/s41562-020-0886-x



Creative Commons License
ISSN: 2195-3325
PsychOpen Logo