Use of Crossed Random-Effects Models to Assess Multiple-Choice Items: An Experimental Study

Jesús Sanz Fernández; Carmen García-Alba; Ricardo Olmos Albacete

Use of Crossed Random-Effects Models to Assess Multiple-Choice Items: An Experimental Study

Sanz, Susana ^[1] ; García, Carmen ^[2] ; Olmos, Ricardo ^[2]
1. [1] Universidad Pontificia Comillas
  
  Universidad Pontificia Comillas
  
  Madrid, España
2. [2] Universidad Autónoma de Madrid
  
  Universidad Autónoma de Madrid
  
  Madrid, España
Localización: The Spanish Journal of Psychology, ISSN 1138-7416, Nº 29, 2026
Idioma: inglés
DOI: 10.1017/sjp.2026.10022
Enlaces
- Texto completo
Resumen
- Abstract Including general options on items, such as None-of-the-above (NOTA), could lead to worse psychometric properties. Further, personality traits are related to academic performance and could influence NOTA effects. Therefore, this study aims to test the effects of NOTA by manipulating its appearance and its use as the correct option or as a distractor, while considering the examinees’ personality traits. The sample consisted of 449 psychology students who answered a statistics concept inventory. A crossed random-effects model was conducted to model the probability of answering an item correctly depending on items and students’ covariates. The appearance of NOTA negatively affects the probability of answering correctly and changes between examinees. Conscientiousness, Extraversion, Emotional Stability, and previous knowledge of statistics are also significant predictors. The results point to increased items’ difficulty when NOTA is included, although not all the examinees seem to show the same behavior when dealing with NOTA options.
Referencias bibliográficas
- Baayen, R. H., Davidson, D. J., & Batesm, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal...
- Beaujean, A. A., Firmin, M. W., Attai, S., Johnson, C. B., Firmin, R. L., & Mena, K. E. (2011). Using personality and cognitive ability...
- Blendermann, M. F., Little, J. L., & Gray, K. M. (2020). How “none of the above” (NOTA) affects the accessibility of tested and related...
- Boland, R. J., Lester, N. A., & Williams, E. (2010). Writing multiple-choice questions. Academic Psychiatry, 34(4), 310–316. https://doi.org/10.1176/appi.ap.34.4.310
- Bryan, E. M., Beadel, J. R., McLeish, A. C., & Teachman, B. A. (2021). Assessment of intolerance of uncertainty: Validation of a modified...
- Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). Guilford Publications.
- Bürkner, P. C. (2021). Bayesian item response modelling in R with brms and Stan. Journal of Statistical Software, 100(5), 1–54. https://doi.org/10.18637/jss.v100.i05
- Butler, A. C. (2018). Multiple-choice testing in education: Are the best practices for assessment also good for learning? Journal of Applied...
- Chamorro-Premuzic, T., & Furnham, A. (2003). Personality traits and academic examination performance. European Journal of Personality,...
- Chamorro-Premuzic, T., & Furnham, A. (2008). Personality, intelligence and approaches to learning as predictors of academic performance....
- Coenen, J., Golsteyn, B. H. H., Stolp, T., & Tempelaar, D. (2021). Personality traits and academic performance: Correcting self-assessed...
- Conard, M. A. (2006). Aptitude is not enough: How personality and behavior predict academic performance. Journal of Research in Personality,...
- Correll, J., Mellinger, C., & Pedersen, E. J. (2022). Flexible approaches for estimating partial eta squared in mixed-effects models with...
- Costa, P., & McCrae, R. (1992). Revised NEO Personality Inventory (NEO-PI-R) and NEO Five-Factor Inventory (NEO-FFI) professional manual....
- De Feyter, T., Caers, R., Vigna, C., & Berings, D. (2012). Unravelling the impact of the Big Five personality traits on academic performance:...
- DiBattista, D., Sinnige-Egger, J., & Fortuna, G. (2014). The “none of the above” option in multiple-choice testing: An experimental study....
- Dochy, F., Moerkerke, G., De Corte, E. D., & Segers, M. (2001). The assessment of quantitative problem-solving skills with “none of the...
- Downing, S. M. (2005). The effects of violating standard item writing principles on test and students: The consequences of using flawed test...
- Feldt, L. S., Woodruff, D. J., & Salih, F. A. (1987). Statistical inference for coefficient alpha. Applied Psychological Measurement,...
- Fisher, G. H. (1973). The linear logistic test model as an instrument in educational research. Acta Psychologica, 37, 359–374. https://doi.org/10.1016/0001-6918(73)90003-6
- Frary, R.B. (1991) The None-of-the-Above Option: An Empirical Study. Applied Measurement in Education, 4(2), 115–124, http://dx.doi.org/10.1207/s15324818ame0402_2
- Frey, B. B., Petersen, S., Edwards, L. M., Pedrotti, J. T., & Peyton, V. (2005). Item-writing rules: Collective wisdom. Teaching and Teacher...
- Furnham, A., & Chamorro-Premuzic, T. (2004). Personality and intelligence as predictors of statistics examination grades. Personality...
- Furnham, A., Chamorro-Premuzic, T., & McDougall, F. (2003). Personality, cognitive ability, and beliefs about intelligence as predictors...
- Furnham, A., & Monsen, J. (2009). Personality traits and intelligence predict academic school grades. Learning and Individual Differences,...
- Hagenmüller, B. (2020). On the impact of the response options’ position on item difficulty in multiple-choice-items. European Journal of Psychological...
- Haladyna, T. M., & Downing, S. M. (1989a). A taxonomy of multiple-choice item-writing rules. Applied Measurement in Education, 2(1), 37–50....
- Haladyna, T. M., & Downing, S. M. (1989b). Validity of a taxonomy of multiple-choice item-writing rules. Applied Measurement in Education,...
- Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment....
- Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. Routledge.10.4324/9780203850381
- Haladyna, T. M., Rodriguez, M. C., & Stevens, C. (2019). Are multiple-choice items too fat? Applied Measurement in Education, 32(4), 350–364....
- Halloun, I., & Hestenes, D. (1985). The initial knowledge state of college physics students. American Journal of Physics, 53(11), 1043–1055....
- Hoffman, L. (2015). Longitudinal Analysis. Modeling withinperson fluctuation and change. Routledge.10.4324/9781315744094
- Hohensinn, C., & Baghaei, P. (2017). Does the position of response options in multiple-choice tests matter? Psicológica, 38(1), 93–109.
- Jang, Y., Pashler, H., & Huber, D. E. (2014). Manipulation of choice familiarity in multiple-choice testing support a retrieval practice...
- Janssen, R., Schepers, J., & Peres, D. (2004). Models with item and item group predictors. In De Boeck, P. & Wilson, M. (Eds.), Explanatory...
- Jöreskog, K. G., & Goldberger, A. S. (1975). Estimation of a model with multiple indicators and multiple causes of a single latent variable....
- Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution...
- Judd, C. M., Westfall, J., & Kenny, D. A. (2017). Experiments with more than one random factor: Designs, analytic models, and statistical...
- Little, J. L. (2023). Does using none-of-the-above (NOTA) hurt students’ confidence? Journal of Intelligence, 11(8), 157. https://doi.org/10.3390/jintelligence11080157
- Martínez, R. J., Moreno, R., Martín, I., & Trigo, M. E. (2008). Evaluation of five guidelines for option development in multiple-choice...
- Muthén, L.-K., & Muthén, B. O. (2017). MPLUS (version 7). [Computer Software]. https://www.statmodel.com/
- Muthén, B. O. (1989). Latent variable modelling in heterogeneous populations. Psychometrika, 54(4), 557–585. https://doi.org/10.1007/BF02296397
- Nye, J. V. C., Orel, E., & Kochergina, E. (2015). Big five personality traits and academic performance in Russian universities. Research...
- Odegard, T.N. & Koen, J.D. (2007) “None of the above” as a correct and incorrect alternative on a multiple-choice test: Implications for...
- Oquendo, M. A., Baca-García, E., Graver, R., Morales, M., Montalvan, V., & Mann, J. J. (2001). Spanish adaptation of the Barratt Impulsiveness...
- Pachai, M. V., DiBattista, D., & Kim, J. A. (2015). A systematic assessment of “None of the above” on a multiple-choice test in a first-year...
- Patton, J. H., Stanford, M. S., & Barrat, E. S. (1995). Factor structure of the Barrat Impulsiveness Scale. Journal of Clinical Psychology,...
- Persson, R. A. X. (2023). Multiple-choice test items with “none of the above” as an alternative. PsyArXiv. https://doi.org/10.31234/osf.io/w7cnr
- R Core Team. (2021). R: A language and environment for statistical computing (version 4.1.2). [Computer Software]. R Foundation for Statistical...
- Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests (Expanded edition, with foreword and afterword by B. D....
- Rich, C. E., & Johanson, G. A. (1990). An Item-Level Analysis of “None of the Above”. Paper presented at the annual meeting of the American...
- Stan Development Team (2021). RStan: The R interface to Stan. R package version 2.21.3. Author.
- Stone, A., Allen, K., Rhoads, T. R., Murphy, T. J., Shehab, R. L., & Saha, C. (2003). The statistic concept inventory: A pilot study....
- Stone, A. (2006). A psychometric analysis of the Statistic Concept Inventory. [Doctoral dissertation]. University of Oklahoma. http://hdl.handle.net/11244/1013
- Tarrant, M., Ware, J., & Mohammed, A. M. (2009). An assessment of functioning and non-functioning distractors in multiple-choice questions:...
- Ulitzsch, E., von Davier, M., & Pohl, S. (2020). A hierarchical latent response model for inferences about examinee engagement in terms...
- Vigil-Colet, A., Morales-Vives, F., Camps, E., Tous, J., & Lorenzo-Seva, U. (2013). Development and validation of Overall Personality...