Measuring Verbal Working Memory Online in Research: Feasibility, Reliability, and Validity of two Implementations

• Federico M. Gonzalez ^[1] ; Jonathan Marrujo ^[1] ; Magalí Martínez ^[1] ; Juan Pablo Barreyro ^[2] ; Débora Burin ^[1]
  1. [1] Universidad de Buenos Aires

     Universidad de Buenos Aires

     Argentina

     
  2. [2] Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET), Argentina
• Localización: Psicológica: Revista de metodología y psicología experimental, ISSN-e 1576-8597, ISSN 0211-2159, Vol. 45, Nº 2, 2024
• Idioma: inglés
• DOI: 10.20350/digitalCSIC/16342
• Enlaces
  • Texto completo
• Resumen

  • Working memory (WM) tasks have been extensively studied with laboratory paper and pencil or computerized tasks; few studies have assessed the feasibility and psychometric properties of WM online tests for low-stakes research purposes.We analyzed two different remote online testing implementations of two verbal WM tasks, Letter-Number Sequencing (LNS) and Running Span (RS).LNS was tested in a supervised, online video conference setting in small groups (n = 115), and RS in an unsupervised, asynchronous, remote assessment (n = 289). Both had adequate reliability. However, in LNS, following criteria for minimum performance and “gaming the test” behavior, around 25% participants had to be discarded; scores for set size 2 did not follow the set size effect; and performance was significantly better for the more difficult set sizes as compared to an in-person similar assessment. Conversely, the remote online RS task showed an expected set size effect, and performance was generally equivalent to a similar in-person assessment. Our results put into question the synchronous video conference implementation of the Letter Number Sequence and favor the asynchronous Running Span task.

• Referencias bibliográficas
  • Alloway, T. P., Moulder, R., Horton, J. C., Leedy, A., Archibald, L. M. D., Burin, D., Injoque-Ricle, I., Passolunghi, M. C., & Dos Santos,...
  • Backx, R., Skirrow, C., Dente, P., Barnett, J. H., & Cormack, F. K. (2020). Comparing web-based and labbased cognitive assessment using...
  • Baddeley, A. (2010). Working memory. Current Biology, 20(4), R136-R140. https://doi.org/10.1016/j.cub.2009.12.014
  • Baddeley, A. D., & Hitch, G. (1974). Working memory. In G.H. Bower (Ed.), The psychology of learning and motivation: advances in research...
  • Baddeley, A., Hitch, G., & Allen, R. (2020). A multicomponent model of working memory. In R. Logie, V. Camos, & N. Cowan (Eds.), Working...
  • Barreyro, J.P.; Injoque-Ricle, R.; Formoso, J. y Burin, D. (2015a). Validez y confiabilidad de la prueba Running Memory Span. Revista Argentina...
  • Barreyro, J. P.; Injoque-Ricle, I.; González, J. M.; Burin; D. I. (2015b). Estudio acerca de las propiedades psicométricas de pruebas clásicas...
  • Barrouillet, P., & Camos, V. (2020). The time-based resource-sharing model of working memory. In R. Logie, V. Camos, & N. Cowan (Eds.),...
  • Bates, D. (2010). lme4: Mixed-effects modeling with R. https://lme4.r-forge.rproject.org/book/front.pdf
  • Brooks, M., Kristensen, K., van Benthem, K., Magnusson, A., Berg, C., Nielsen, A., Skaug, H., Maechler, M., & Bolker, B. (2017). glmmTMB...
  • Cowan, N. (2017). The many faces of working memory and short-term storage. Psychonomic Bulletin & Review, 24(4), 1158-1170. https://doi.org/10.3758/s13423-016-1191-6
  • Cowan, N., Morey, C. C., & Naveh-Benjamin, M. (2020). An embedded-processes approach to working memory: How is it distinct from other...
  • Crivelli, L., Quiroz, Y. T., Calandri, I. L., Martin, M. E., Velilla, L. M., Cusicanqui, M. I., Yglesias, F. C., LlibreRodríguez, J. J., Armele,...
  • Do, B.-R. (2009). Research on unproctored internet testing. Industrial and Organizational Psychology, 2, 49–51. https://doi.org/10.1111/j.1754-9434.2008.01107.x
  • Emery, L., Myerson, J., & Hale, S. (2007). Age differences in item manipulation span: The case of letternumber sequencing. Psychology...
  • Feldman Barrett, L., Tugade, M. M., & Engle, R. W. (2004). Individual differences in working memory capacity and dual-process theories...
  • Gosling, S. D., & Mason, W. (2015). Internet research in psychology. Annual Review of Psychology, 66, 877–902. https://doi.org/10.1146/annurev-psych-010814-015321
  • Hauser, D.J., Moss, A.J., Rosenzweig, C., Jaffe, S. N., Robinson, J. & Litman, L. (2022). Evaluating CloudResearch’s Approved Group as...
  • Henrich, J., Heine, S. J., & Norenzayan, A. (2010). Most people are not WEIRD. Nature, 466(7302), 29-29. https://doi.org/10.1038/466029a
  • Hicks, K. L., Foster, J. L., & Engle, R. W. (2016). Measuring working memory capacity on the web with the online working memory lab (the...
  • International Test Commision (2022). Guidelines for technology-based assessment (Draft). https://www.intestcom.org/upload/media-library/tba-guidelines-3-14-2022-draftnumbered1647343978NGDYR.pdf
  • Jaroslawska, A. J., & Rhodes, S. (2019). Adult age differences in the effects of processing on storage in working memory: A meta-analysis....
  • Kane, M. J., Hambrick, D. Z., Tuholski, S. W., Wilhelm, O., Payne, T. W., & Engle, R. W. (2004). The generality of working memory capacity:...
  • Kulikowski, K., & Potasz-Kulikowska, K. (2016). Can we measure working memory via the internet? the reliability and factorial validity...
  • Leidheiser, W., Branyon, J., Baldwin, N., Pak, R., & McLaughlin, A. (2015). Lessons learned in adapting a lab-based measure of working...
  • Length, R. (2021). emmeans: Estimated marginal means, aka least-squares means. https://CRAN.Rproject.org/package=emmeans
  • Leong, V., Raheel, K., Sim, J. Y., Kacker, K., Karlaftis, V. M., Vassiliu, C., Kalaivanan, K., Chen, S. H. A., Robbins, T. W., Sahakian, B....
  • Logie, R., Camos, V., & Cowan, N. (Eds.) (2020). Working memory: the state of the science. Oxford University Press. https://doi.org/10.1093/oso/9780198842286.001.0001
  • Logie, R. H. (2016). Retiring the central executive. Quarterly Journal of Experimental Psychology, 69(10), 2093-2109. https://doi.org/10.1080/17470218.2015.1136657
  • Lüdecke D (2023). sjPlot: Data Visualization for Statistics in Social Science. R package version 2.8.15, https://CRAN.R-project.org/package=sjPlot.
  • Macnamara, B. N., & Conway, A. R. A. (2016). Working memory capacity as a predictor of simultaneous language interpreting performance....
  • Mashburn, C. A., Tsukahara, J. S., & Engle, R. W. (2020). Individual differences in attention control: Implications for the relationship...
  • Mielicki, M. K., Koppel, R. H., Valencia, G., & Wiley, J. (2018). Measuring working memory capacity with the letter–number sequencing...
  • Miyake, A., & Shah, P. (Eds.) (1999a). Models of working memory: mechanisms of active maintenance and executive control. Cambridge University...
  • Miyake, A., & Shah, P. (1999b). Toward unified theories of working memory: emerging general consensus, unresolved theoretical issues,...
  • Oberauer, K. (2020). Towards a theory of working memory: from metaphors to mechanisms. In R. Logie, V. Camos, & N. Cowan (Eds.), Working...
  • Oberauer, K., Lewandowsky, S., Awh, E., Brown, G. D. A., Conway, A., Cowan, N., Donkin, C., Farrell, S., Hitch, G. J., Hurlstone, M. J., Ma,...
  • R Core Team (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing. URL https://www.R-project.org/Reips,...
  • Revelle, W. (2021) Psych: procedures for personality and psychological research. Northwestern University, Evanston, Illinois, USA, https://CRAN.R-project.org/package=psych...
  • Rhodes, S., Doherty, J. M., Jaroslawska, A. J., Forsberg, A., Belletier, C., Naveh-Benjamin, M., Cowan, N., Barrouillet, P., Camos, V., &...
  • RStudio Team (2023). RStudio: integrated development environment for R. RStudio, URL http://www.rstudio.com/
  • Ruiz, S., Chen, X., Rebuschat, P., & Meurers, D. (2019) Measuring individual differences in cognitive abilities in the lab and on the...
  • Shelton, J. T., Metzger, R. L., & Elliott, E. M. (2007). A group-administered lag task as a measure of working memory. Behavior Research...
  • Steger, D., Schroeders, U., & Gnambs, T. (2020). A meta-analysis of test scores in proctored and unproctored ability assessments. European...
  • Stoet, G. (2010). Psytoolkit: A software package for programming psychological experiments using Linux. Behavior Research Methods, 42(4),...
  • Stoet, G. (2017). Psytoolkit: a novel web-based method for running online questionnaires and reactiontime experiments. Teaching of Psychology,...
  • Webb, M. A., & Tangney, J. P. (2023). Too good to be true: bots and bad data from Mechanical Turk. Perspectives on Psychological Science...
  • Wechsler, D. (2003). WAIS III: test de inteligencia para adultos. Paidós.
  • Wickham H, Averick M, Bryan J, Chang W, McGowan LD, François R, Grolemund G, Hayes A, Henry L, Hester J, Kuhn M, Pedersen TL, Miller E, Bache...
  • Lakens, D. (2022). Improving Your Statistical Inferences. https://lakens.github.io/statistical_inferences/. https://doi.org/10.5281/zenodo.6409077
  • Green, P. & MacLeod, C.J. (2016), SIMR: an R package for power analysis of generalized linear mixed models by simulation. Methods in Ecology...
  • Miller, J.E. (2008). Interpreting the substantive significance of multivariable regression coefficients. Proceedings of the American Statistical...