Botvinik-Nezer, Rotem ORCID: https://orcid.org/0000-0003-2669-1877, Holzmeister, Felix ORCID: https://orcid.org/0000-0001-9606-0427, Camerer, Colin F. ORCID: https://orcid.org/0000-0003-4049-1871, Dreber, Anna ORCID: https://orcid.org/0000-0003-3989-9941, Huber, Juergen, Johannesson, Magnus ORCID: https://orcid.org/0000-0001-8759-6393, Kirchler, Michael ORCID: https://orcid.org/0000-0002-5416-2545, Iwanir, Roni, Mumford, Jeanette A., Adcock, R. Alison ORCID: https://orcid.org/0000-0002-7859-5484, Avesani, Paolo ORCID: https://orcid.org/0000-0001-8943-8911, Baczkowski, Blazej M. ORCID: https://orcid.org/0000-0001-9827-797X, Bajracharya, Aahana ORCID: https://orcid.org/0000-0002-7361-6020, Bakst, Leah ORCID: https://orcid.org/0000-0003-2741-5532, Ball, Sheryl, Barilari, Marco ORCID: https://orcid.org/0000-0002-3313-3120, Bault, Nadège ORCID: https://orcid.org/0000-0001-8244-6062, Beaton, Derek ORCID: https://orcid.org/0000-0001-6118-4366, Beitner, Julia ORCID: https://orcid.org/0000-0002-2539-7011, Benoit, Roland G. ORCID: https://orcid.org/0000-0002-7127-4899, Berkers, Ruud M. W. J. ORCID: https://orcid.org/0000-0002-0347-2375, Bhanji, Jamil P. ORCID: https://orcid.org/0000-0003-3451-8732, Biswal, Bharat B., Bobadilla-Suarez, Sebastian ORCID: https://orcid.org/0000-0002-5951-0772, Bortolini, Tiago ORCID: https://orcid.org/0000-0003-2097-5537, Bottenhorn, Katherine L. ORCID: https://orcid.org/0000-0002-7796-8795, Bowring, Alexander ORCID: https://orcid.org/0000-0002-5553-7249, Braem, Senne, Brooks, Hayley R. ORCID: https://orcid.org/0000-0001-5371-2624, Brudner, Emily G. ORCID: https://orcid.org/0000-0002-9981-3807, Calderon, Cristian B., Camilleri, Julia A., Castrellon, Jaime J. ORCID: https://orcid.org/0000-0001-5834-7101, Cecchetti, Luca ORCID: https://orcid.org/0000-0001-5184-6477, Cieslik, Edna C. ORCID: https://orcid.org/0000-0001-5491-5837, Cole, Zachary J. ORCID: https://orcid.org/0000-0002-0692-5739, Collignon, Olivier ORCID: https://orcid.org/0000-0003-1882-3550, Cox, Robert W. ORCID: https://orcid.org/0000-0002-0031-1568, Cunningham, William A. ORCID: https://orcid.org/0000-0002-7063-5859, Czoschke, Stefan ORCID: https://orcid.org/0000-0001-7863-8272, Dadi, Kamalaker, Davis, Charles P. ORCID: https://orcid.org/0000-0002-7293-2769, Luca, Alberto De ORCID: https://orcid.org/0000-0002-2553-7299, Delgado, Mauricio R., Demetriou, Lysia ORCID: https://orcid.org/0000-0001-5249-0900, Dennison, Jeffrey B. ORCID: https://orcid.org/0000-0003-0557-3548, Di, Xin ORCID: https://orcid.org/0000-0002-2422-9016, Dickie, Erin W. ORCID: https://orcid.org/0000-0003-3028-9864, Dobryakova, Ekaterina ORCID: https://orcid.org/0000-0002-6005-2115, Donnat, Claire L. ORCID: https://orcid.org/0000-0001-7079-8060, Dukart, Juergen ORCID: https://orcid.org/0000-0003-0492-5644, Duncan, Niall W. ORCID: https://orcid.org/0000-0002-3225-9119, Durnez, Joke ORCID: https://orcid.org/0000-0001-9030-2202, Eed, Amr ORCID: https://orcid.org/0000-0002-9370-4917, Eickhoff, Simon B., Erhart, Andrew ORCID: https://orcid.org/0000-0001-5195-0880, Fontanesi, Laura ORCID: https://orcid.org/0000-0001-5067-9451, Fricke, G. Matthew ORCID: https://orcid.org/0000-0002-4466-8649, Fu, Shiguang ORCID: https://orcid.org/0000-0002-1313-0176, Galván, Adriana, Gau, Remi ORCID: https://orcid.org/0000-0002-1535-9767, Genon, Sarah ORCID: https://orcid.org/0000-0002-7087-7882, Glatard, Tristan ORCID: https://orcid.org/0000-0003-2620-5883, Glerean, Enrico ORCID: https://orcid.org/0000-0003-0624-675X, Goeman, Jelle J. ORCID: https://orcid.org/0000-0003-4283-0259, Golowin, Sergej A. E., González-García, Carlos ORCID: https://orcid.org/0000-0001-6627-5777, Gorgolewski, Krzysztof J. ORCID: https://orcid.org/0000-0003-3321-7583, Grady, Cheryl L., Green, Mikella A. ORCID: https://orcid.org/0000-0002-8896-8520, Guassi Moreira, João F. ORCID: https://orcid.org/0000-0001-8312-4487, Guest, Olivia ORCID: https://orcid.org/0000-0002-1891-0972, Hakimi, Shabnam ORCID: https://orcid.org/0000-0003-4122-6041, Hamilton, J. Paul ORCID: https://orcid.org/0000-0001-7982-4659, Hancock, Roeland ORCID: https://orcid.org/0000-0001-8932-6872, Handjaras, Giacomo ORCID: https://orcid.org/0000-0001-5677-8434, Harry, Bronson B. ORCID: https://orcid.org/0000-0003-0389-0945, Hawco, Colin, Herholz, Peer ORCID: https://orcid.org/0000-0002-9840-6257, Herman, Gabrielle ORCID: https://orcid.org/0000-0003-4695-5251, Heunis, Stephan ORCID: https://orcid.org/0000-0003-3503-9872, Hoffstaedter, Felix, Hogeveen, Jeremy ORCID: https://orcid.org/0000-0001-9612-2085, Holmes, Susan ORCID: https://orcid.org/0000-0002-2208-8168, Hu, Chuan-Peng ORCID: https://orcid.org/0000-0002-7503-5131, Huettel, Scott A. ORCID: https://orcid.org/0000-0002-5092-4936, Hughes, Matthew E. ORCID: https://orcid.org/0000-0001-5057-5556, Iacovella, Vittorio ORCID: https://orcid.org/0000-0002-0853-1573, Iordan, Alexandru D. ORCID: https://orcid.org/0000-0002-1391-785X, Isager, Peder M. ORCID: https://orcid.org/0000-0002-6922-3590, Isik, Ayse I. ORCID: https://orcid.org/0000-0002-1652-9297, Jahn, Andrew ORCID: https://orcid.org/0000-0001-5286-049X, Johnson, Matthew R. ORCID: https://orcid.org/0000-0003-3953-8898, Johnstone, Tom ORCID: https://orcid.org/0000-0001-8635-8158, Joseph, Michael J. E. ORCID: https://orcid.org/0000-0002-0068-230X, Juliano, Anthony C. ORCID: https://orcid.org/0000-0001-7433-6891, Kable, Joseph W. ORCID: https://orcid.org/0000-0001-6495-9040, Kassinopoulos, Michalis ORCID: https://orcid.org/0000-0003-4312-4401, Koba, Cemal ORCID: https://orcid.org/0000-0001-7097-1441, Kong, Xiang-Zhen ORCID: https://orcid.org/0000-0002-0805-1350, Koscik, Timothy R. ORCID: https://orcid.org/0000-0002-2551-8127, Kucukboyaci, Nuri Erkut ORCID: https://orcid.org/0000-0002-7613-641X, Kuhl, Brice A. ORCID: https://orcid.org/0000-0001-5229-5400, Kupek, Sebastian, Laird, Angela R. ORCID: https://orcid.org/0000-0003-3379-8744, Lamm, Claus ORCID: https://orcid.org/0000-0002-5422-0653, Langner, Robert ORCID: https://orcid.org/0000-0002-3237-001X, Lauharatanahirun, Nina, Lee, Hongmi ORCID: https://orcid.org/0000-0001-8023-0727, Lee, Sangil ORCID: https://orcid.org/0000-0002-4443-9926, Leemans, Alexander ORCID: https://orcid.org/0000-0002-9306-6126, Leo, Andrea ORCID: https://orcid.org/0000-0002-1948-6608, Lesage, Elise ORCID: https://orcid.org/0000-0002-5517-6965, Li, Flora ORCID: https://orcid.org/0000-0002-9367-2257, Li, Monica Y. C. ORCID: https://orcid.org/0000-0003-1775-4434, Lim, Phui Cheng ORCID: https://orcid.org/0000-0002-5709-8956, Lintz, Evan N. ORCID: https://orcid.org/0000-0003-4077-7749, Liphardt, Schuyler W. ORCID: https://orcid.org/0000-0001-8370-8722, Losecaat Vermeer, Annabel B. ORCID: https://orcid.org/0000-0001-5631-0233, Love, Bradley C. ORCID: https://orcid.org/0000-0002-7883-7076, Mack, Michael L. ORCID: https://orcid.org/0000-0001-5155-7413, Malpica, Norberto ORCID: https://orcid.org/0000-0003-4618-7459, Marins, Theo ORCID: https://orcid.org/0000-0002-5017-707X, Maumet, Camille ORCID: https://orcid.org/0000-0002-6290-553X, McDonald, Kelsey ORCID: https://orcid.org/0000-0002-7484-5300, McGuire, Joseph T. ORCID: https://orcid.org/0000-0001-6259-0809, Melero, Helena ORCID: https://orcid.org/0000-0002-5373-6711, Méndez Leal, Adriana S. ORCID: https://orcid.org/0000-0002-0147-9450, Meyer, Benjamin, Meyer, Kristin N. ORCID: https://orcid.org/0000-0003-0190-7678, Mihai, Glad ORCID: https://orcid.org/0000-0001-5715-6442, Mitsis, Georgios D. ORCID: https://orcid.org/0000-0001-9975-5128, Moll, Jorge, Nielson, Dylan M. ORCID: https://orcid.org/0000-0003-4613-6643, Nilsonne, Gustav ORCID: https://orcid.org/0000-0001-5273-0150, Notter, Michael P. ORCID: https://orcid.org/0000-0002-5866-047X, Olivetti, Emanuele ORCID: https://orcid.org/0000-0001-7653-1519, Onicas, Adrian I. ORCID: https://orcid.org/0000-0002-5953-8548, Papale, Paolo ORCID: https://orcid.org/0000-0002-6249-841X, Patil, Kaustubh R. ORCID: https://orcid.org/0000-0002-0289-5480, Peelle, Jonathan E. ORCID: https://orcid.org/0000-0001-9194-854X, Pérez, Alexandre ORCID: https://orcid.org/0000-0003-0556-0763, Pischedda, Doris ORCID: https://orcid.org/0000-0003-4694-2493, Poline, Jean-Baptiste ORCID: https://orcid.org/0000-0002-9794-749X, Prystauka, Yanina ORCID: https://orcid.org/0000-0001-8258-2339, Ray, Shruti ORCID: https://orcid.org/0000-0002-1864-386X, Reuter-Lorenz, Patricia A. ORCID: https://orcid.org/0000-0002-6977-3429, Reynolds, Richard C. ORCID: https://orcid.org/0000-0002-7267-5563, Ricciardi, Emiliano ORCID: https://orcid.org/0000-0002-7178-9534, Rieck, Jenny R. ORCID: https://orcid.org/0000-0002-3242-6407, Rodriguez-Thompson, Anais M. ORCID: https://orcid.org/0000-0002-3926-8958, Romyn, Anthony ORCID: https://orcid.org/0000-0002-6333-7888, Salo, Taylor ORCID: https://orcid.org/0000-0001-9813-3167, Samanez-Larkin, Gregory R. ORCID: https://orcid.org/0000-0001-7846-3804, Sanz-Morales, Emilio ORCID: https://orcid.org/0000-0001-8066-9940, Schlichting, Margaret L. ORCID: https://orcid.org/0000-0002-5433-8870, Schultz, Douglas H. ORCID: https://orcid.org/0000-0003-0809-9036, Shen, Qiang ORCID: https://orcid.org/0000-0003-1330-1315, Sheridan, Margaret A., Silvers, Jennifer A. ORCID: https://orcid.org/0000-0002-6615-5473, Skagerlund, Kenny ORCID: https://orcid.org/0000-0003-1894-3595, Smith, Alec ORCID: https://orcid.org/0000-0002-1938-4660, Smith, David V. ORCID: https://orcid.org/0000-0001-5754-9633, Sokol-Hessner, Peter ORCID: https://orcid.org/0000-0003-0895-5576, Steinkamp, Simon R. ORCID: https://orcid.org/0000-0002-6437-0700, Tashjian, Sarah M. ORCID: https://orcid.org/0000-0002-0946-6662, Thirion, Bertrand ORCID: https://orcid.org/0000-0001-5018-7895, Thorp, John N. ORCID: https://orcid.org/0000-0002-9307-6690, Tinghög, Gustav ORCID: https://orcid.org/0000-0002-8159-1249, Tisdall, Loreen ORCID: https://orcid.org/0000-0001-6303-6148, Tompson, Steven H. ORCID: https://orcid.org/0000-0002-6315-1485, Toro-Serey, Claudio ORCID: https://orcid.org/0000-0002-5273-2276, Torre Tresols, Juan Jesus, Tozzi, Leonardo ORCID: https://orcid.org/0000-0002-9429-6476, Truong, Vuong ORCID: https://orcid.org/0000-0003-1412-498X, Turella, Luca ORCID: https://orcid.org/0000-0003-1096-2150, van ‘t Veer, Anna E. ORCID: https://orcid.org/0000-0002-2733-1841, Verguts, Tom ORCID: https://orcid.org/0000-0002-7783-4754, Vettel, Jean M. ORCID: https://orcid.org/0000-0001-5336-2986, Vijayarajah, Sagana, Vo, Khoi ORCID: https://orcid.org/0000-0002-6000-2628, Wall, Matthew B. ORCID: https://orcid.org/0000-0002-0493-6274, Weeda, Wouter D. ORCID: https://orcid.org/0000-0001-8619-2645, Weis, Susanne ORCID: https://orcid.org/0000-0001-7726-6470, White, David J. ORCID: https://orcid.org/0000-0001-8694-1474, Wisniewski, David ORCID: https://orcid.org/0000-0003-4793-1171, Xifra-Porxas, Alba ORCID: https://orcid.org/0000-0002-9023-2432, Yearling, Emily A., Yoon, Sangsuk ORCID: https://orcid.org/0000-0002-3399-1096, Yuan, Rui ORCID: https://orcid.org/0000-0003-4800-3895, Yuen, Kenneth S. L. ORCID: https://orcid.org/0000-0002-9465-9070, Zhang, Lei ORCID: https://orcid.org/0000-0002-9586-595X, Zhang, Xu ORCID: https://orcid.org/0000-0003-1988-7767, Zosky, Joshua E. ORCID: https://orcid.org/0000-0002-1962-3633, Nichols, Thomas E. ORCID: https://orcid.org/0000-0002-4516-5103, Poldrack, Russell A. ORCID: https://orcid.org/0000-0001-6755-0259 and Schonberg, Tom ORCID: https://orcid.org/0000-0002-4485-816X (2020) Variability in the analysis of a single neuroimaging dataset by many teams. Nature, 582 (7810). pp. 84-88. ISSN 0028-0836 (Print), 1476-4687 (Online) (doi:10.1038/s41586-020-2314-9)

Preview	PDF (Author Accepted Manuscript) 29488 XIFRA-PORXAS_Variability_in_the_Analysis_of_a_Single_Neuroimaging_Dataset_2020.pdf - Accepted Version Download (3MB) | Preview
Preview	PDF (Supplementary Information) 29488 XIFRA-PORXAS_Supplementary_Information_2020.pdf - Supplemental Material Download (210kB) | Preview

Abstract

Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

Item Type:	Article
Uncontrolled Keywords:	fMRI, analytic variability
Subjects:	B Philosophy. Psychology. Religion > BF Psychology
Faculty / School / Research Centre / Research Group:	Faculty of Education, Health & Human Sciences Faculty of Education, Health & Human Sciences > School of Human Sciences (HUM)
Last Modified:	21 Nov 2020 01:38
URI:	http://gala.gre.ac.uk/id/eprint/29488

Actions (login required)

View Item

Downloads