Farkas, Bence, Krajcsi, Attila, Janacsek, Karolina ORCID: 0000-0001-7829-8220 and Nemeth, Dezso (2023) The complexity of measuring reliability in learning tasks: an illustration using the Alternating Serial Reaction Time task. Behavior Research Methods. pp. 1-17. ISSN 1554-351X (Print), 1554-3528 (Online) (doi:https://doi.org/10.3758/s13428-022-02038-5)

Preview

PDF (Publisher VoR) 38507_JANACSEK_The_complexity_of_measuring_reliability_in_learning_tasks.pdf - Published Version Available under License Creative Commons Attribution. Download (2MB) | Preview

Abstract

Despite the fact that reliability estimation is crucial for robust inference, it is underutilized in neuroscience and cognitive psychology. Appreciating reliability can help researchers increase statistical power, effect sizes, and reproducibility, decrease the impact of measurement error, and inform methodological choices. However, accurately calculating reliability for many experimental learning tasks is challenging. In this study, we highlight a number of these issues, and estimate multiple metrics of internal consistency and split-half reliability of a widely used learning task on a large sample of 180 subjects. We show how pre-processing choices, task length, and sample size can affect reliability and its estimation. Our results show that the Alternating Serial Reaction Time Task has respectable reliability, especially when learning scores are calculated based on reaction times and two-stage averaging. We also show that a task length of 25 blocks can be sufficient to meet the usual thresholds for minimally acceptable reliability. We further illustrate how relying on a single point estimate of reliability can be misleading, and the calculation of multiple metrics, along with their uncertainties, can lead to a more complete characterization of the psychometric properties of tasks.

Item Type:	Article
Uncontrolled Keywords:	split-half reliability; Cronbach alpha; bootstrapping; learning tasks; sequence learning; internal validity
Subjects:	B Philosophy. Psychology. Religion > BF Psychology L Education > L Education (General) R Medicine > RC Internal medicine > RC0321 Neuroscience. Biological psychiatry. Neuropsychiatry
Faculty / School / Research Centre / Research Group:	Faculty of Education, Health & Human Sciences Faculty of Education, Health & Human Sciences > Institute for Lifecourse Development Faculty of Education, Health & Human Sciences > Institute for Lifecourse Development > Centre for Thinking and Learning Faculty of Education, Health & Human Sciences > School of Human Sciences (HUM)
Last Modified:	06 Feb 2023 11:37
URI:	http://gala.gre.ac.uk/id/eprint/38507

Actions (login required)

View Item

Downloads