Barnabas, Szaszi, Balazs, Aczel, Harry, Clelland ORCID: https://orcid.org/0000-0001-7292-7129, Marton, Kovacs, Felix, Holzmeister, Don, van Ravenzwaaij, Hannah, Schulz-Kümpel, Sabine, Hoffmann, Gustav, Nilsonne, Livia, Kosa, Zoltan, A. Torma, Yousuf, Abdelfatah, Christopher, Aberson, Oguz A., Acar, Ensar, Acem, Matus, Adamkovic, Timofey, Adamovich, Krisna, Adiasto, Love, Ahnström, Atakan M., Akil, Adil, Al-Busaidi, Ali H., Al-Hoorie, Casper J., Albers, Peter J., Allen, Taym, Alsalti, Micah, Altman, Shilaan, Alzahawi, Ettore, Ambrosini, Saule, Anafinova, Rahul, Anand, Martin, Angerer, Ariadna, Angulo-Brunet, Alberto, Antonietti, Jozsef, Arato, Andreu, Arenas, Marco M., Aviña, Flavio, Azevedo, Marko, Bachl, Bence, Bago, Štěpán, Bahník, Bradley J., Baker, Elza, Balayan, Cassandra, Baldwin, Benjamin, Banai, Kasia, Banas, František, Bartoš, Ernest, Baskin, Jojanneke A., Bastiaansen, Nadège, Bault, Christopher W., Bauman, Quintin, Beazer, Maciej, Behnke, Theiss, Bendixen, Sebastian, Berger, Anna, Bernard, Ursa, Bernardic, Paul A., Bloom, Annika, Boldt, Ciril, Bosch-Rosa, Rotem, Botvinik-Nezer, Adam, Bouyamourn, Ozge, Bozkurt, Laurel, Brehm, Johannes, Breuer, Ryan, Briggs, Hilmar, Brohmer, Testori, Martina ORCID: https://orcid.org/0000-0001-7292-7129 and et, al. (2025) Investigating the analytical robustness of the social and behavioural sciences. Nature. ISSN 0028-0836 (Print), 1476-4687 (Online) (In Press)

PDF (Author's Accepted Manuscript) 51709 TESTORI_Investigating_The_Analytical_Robustness_Of_The_Social_And_Behavioural_Sciences_(AAM)_2025.pdf - Accepted Version Restricted to Repository staff only Download (828kB) | Request a copy

Abstract

The same dataset can be analysed in different justifiable ways to answer the same research question, potentially challenging the robustness of empirical science. In this crowd initiative, we investigated the degree to which research findings in the social and behavioural sciences are contingent on analysts’ choices. We examined a stratified random sample of 100 studies published between 2009 and 2018, where for one claim per study, at least five re-analysts independently re-analysed the original data. The statistical appropriateness of the re-analyses was assessed in peer evaluations, and the robustness indicators were inspected along a range of research characteristics and study designs. We found that 34% of the independent re-analyses yielded the same result (within a tolerance region of +/- 0.05 Cohen’s d) as the original report; with a four times broader tolerance region, this indicator rose to 57%. Regarding the conclusions drawn, 74% of analyses were reported to arrive at the same conclusion as in the original investigation; 24% to no effects/inconclusive result, and 2% to the opposite effect as in the original investigation. This exploratory study suggests that the common single-path analyses in social and behavioural research should not simply be assumed to be robust to alternative analyses4. Therefore, we recommend the development and use of practices to explore and communicate this neglected source of uncertainty.

Item Type:	Article
Uncontrolled Keywords:	analytical variability, crowdsourcing science, data analysis, research credibility, robustness, scientific transparency
Subjects:	H Social Sciences > H Social Sciences (General)
Faculty / School / Research Centre / Research Group:	Greenwich Business School Greenwich Business School > Networks and Urban Systems Centre (NUSC) Greenwich Business School > School of Business, Operations and Strategy
Last Modified:	24 Nov 2025 17:08
URI:	https://gala.gre.ac.uk/id/eprint/51709

Actions (login required)

View Item

Downloads