FoCo: a Shiny app for formative assessment using cognitive diagnosis modeling

• Susana Sanz ^[1] ; Rodrigo S. Kreitchmann ^[1] ; Pablo Nájera ^[1] ; José David Moreno ^[1] ; José Ángel Martínez-Huertas ^[2] ; Miguel A. Sorrel ^[1]
  1. [1] Universidad Autónoma de Madrid

     Universidad Autónoma de Madrid

     Madrid, España

     
  2. [2] Universidad Nacional de Educación a Distancia

     Universidad Nacional de Educación a Distancia

     Madrid, España

     
• Localización: Psicología educativa, ISSN 1135-755X, Vol. 29, Nº. 2, 2023, págs. 149-158
• Idioma: inglés
• DOI: 10.5093/psed2022a14
• Títulos paralelos:
  • FoCo: una aplicación Shiny para la evaluación formativa usando modelos de diagnóstico cognitivo
• Enlaces
  • Texto completo (pdf)
• Resumen
  • español

    La combinación de evaluaciones formativas y sumativas podría mejorar la evaluación. El modelado de diagnóstico cognitivo (MDC) se ha propuesto para diagnosticar fortalezas y debilidades de estudiantes en la evaluación formativa. Sin embargo, ningún software permite implementarlo fácilmente. Así, se ha desarrollado FoCo (https://foco.shinyapps.io/FoCo/), permitiendo realizar análisis MDC y teoría clásica de tests. Se analizaron respuestas de 86 estudiantes de grado a un examen de métodos de investigación, diagnosticándose sus fortalezas y necesidades en cuanto a su dominio de los contenidos de la asignatura y las tres primeras competencias de la taxonomía de Bloom y se analizó la validez de los resultados. El análisis ha sido informativo, ya que para estudiantes con puntuaciones similares ha sido posible detectar diferentes fortalezas y debilidades. Además, se encontró que estos atributos predicen criterios relevantes. Se espera que FoCo facilite el uso de MDC en contextos educativos.

  • English

    Combining formative and summative evaluations could improve assessment. Cognitive diagnosis modeling (CDM) has been proposed as a tool for diagnosing students’ strengths and weaknesses in formative assessment. However, there is no user-friendly software to implement it. For this reason, a Shiny app, FoCo, has been developed (https://foco.shinyapps.io/FoCo/), to conduct CDM and classical test theory analyses. The responses from 86 undergraduate students to a research methods course examination were analyzed. Students’ strengths and needs were diagnosed concerning their dominance of the syllabus contents and the first three competencies in Bloom’s taxonomy. The validity of the results was analyzed. The exam showed acceptable about evaluating students’ knowledge, as students with similar scores showed different strengths and weaknesses. Additionally, these attributes were found to predict different relevant criteria. It is expected that FoCo’s easiness to use promotes the employment of CDM in real educational settings.

• Referencias bibliográficas
  • Abad, F. J., Olea, J., Ponsoda, V., & García, C. (2011). Medición en ciencias sociales y de la salud [Measurement in social and educational...
  • Akbay, L., & de la Torre, J. (2020). Estimation approaches in cognitive diagnosis modeling when attributes are hierarchically structured....
  • Attali, Y., & Fraenkel, T. (2000). The point-biserial as a discrimination index for distractors in multiple-choice items: Deficiencies...
  • Bennett, R. E. (2011). Formative assessment: A critical review. Assessment in Education: Principles, Policy & Practice, 18(1), 5-25. https://doi.org/10.1...
  • Bloom, B., Engelhart, M., Furst, E., Hill, W., & Krathwohl, D. (1956). Taxonomy of educational objectives: The classification of educational...
  • Bradshaw, L., Izsák, A., Templin, J., & Jacobson, E. (2014). Diagnosing teachers’ understandings of rational numbers: Building a multidimensional...
  • Cerezo, R., Fernández, E., Amieiro, N., Valle, A., Rosário, P., & Núñez, J. C. (2019). Mediating role of self-efficacy and usefulness...
  • Chang, W., Cheng, J., Allaire, J. J., Sievert, C., Schloerke, B., Xie, Y., Allen, J., McPherson, J., Dipert, A., & Borges, B. (2021)....
  • Chen, H., & Chen, J. (2016). Retrofitting non-cognitive-diagnostic reading assessment under the generalized DINA model framework. Language...
  • Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555
  • Chiu, C.Y., Sun, Y., & Bian, Y. (2018). Cognitive diagnosis for small educational programs: The general nonparametric classification method....
  • De la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199. https://doi.org/10.1007/s11336-011-9207-7
  • De la Torre, J., & Minchen, N. (2014). Cognitively diagnostic assessments and the cognitive diagnosis model framework. Psicología Educativa,...
  • Fan, T., Song, J., & Guan, Z. (2021). Integrating diagnostic assessment into curriculum: A theoretical framework and teaching practices....
  • Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382. https://doi.org/10.1037/...
  • Fraile, J., Pardo, R., & Panadero, E. (2017). ¿Cómo emplear las rúbricas para implementar una verdadera evaluación formativa? Revista...
  • García-Martín, J., & García-Sánchez, J. N. (2018). The instructional effectiveness of two virtual approaches: processes and product [La...
  • Green, S. B., & Yang, Y. (2009). Reliability of summed item scores using structural equation modeling: An alternative to coefficient alpha....
  • George, D., & Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference. 11.0 update (4th ed.). Allyn & Bacon.
  • Jang, E. E., Dunlop, M., Park, G., & van der Boom, E. H. (2015). How do young students with different profiles of reading skill mastery,...
  • Johnson, M. S., & Sinharay, S. (2020). The reliability of the posterior probability of skill attainment in diagnostic classification models....
  • Jurich, D. P., & Bradshaw, L. P. (2014). An illustration of diagnostic classification modeling in student learning outcomes assessment....
  • Katz, M. H. (2006). Multivariate analyses. A practical guide for clinicians. (2nd ed.). Cambridge University Press.
  • Kreitchmann, R. S., de la Torre, J., Sorrel, M. A., Nájera, P., & Abad, F. J. (2022) Improving reliability estimation in cognitive diagnosis...
  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. https://doi....
  • Li, H., & Suen, H. K. (2013). Constructing and validating a Q-matrix for cognitive diagnostic analyses of a reading test. Educational...
  • Linstone, H. A., & Turoff, M. (Eds.). (1975). The Delphi method. Techniques and applications. Addison-Wesley.
  • Liu, Y., Tian, W., & Xin, T. (2016). An application of M2 statistic to evaluate the fit of cognitive diagnostic models. Journal of Educational...
  • Ma, W., Terzi, R., & de la Torre, J. (2021). Detecting differential item functioning using multiple-group cognitive diagnosis models....
  • Marchesi, A. & Martín, E. (2014). Calidad de la enseñanza en tiempos de crisis. Alianza Editorial.
  • Muthén, L. K., & Muthén, B. O. (1998-2017). Mplus user’s guide (8th ed.). Muthén & Muthén.
  • Nájera, P., Abad, F .J., & Sorrel, M. A. (2021). Determining the number of attributes in cognitive diagnosis modeling. Frontiers in Psychology,...
  • Nájera, P., Sorrel, M. A., & Abad, F. J. (2021). cdmTools: Useful tools for cognitive diagnosis modeling. R package version 1.0.0. https://cran.rproject.org/web/packages/cdmTools/
  • Nájera, P., Sorrel, M. A., de la Torre, J., & Abad, F. J. (2021). Balancing fit and parsimony to improve Q-matrix validation. British...
  • Paulsen, J., & Valdivia, D. S. (2021) Examining cognitive diagnostic modeling in classroom assessment conditions. The Journal of Experimental...
  • R Development Core Team (2021). R: A language and environment for statistical computing. version 4.0.4. R Foundation for Statistical Computing,...
  • Ren, H., Xu, N., Lin, Y., Zhang, S., & Yang, T. (2021). Remedial teaching and learning from A cognitive diagnostic model perspective:...
  • Rojas, G., de la Torre, J., & Olea, J. (2012, April). Choosing between general and specific cognitive diagnosis models when the sample...
  • Santos, M. G., & Matas, S.G. (2007). Self-assessing primary competences in Psychology students [La autoevaluación de las competencias...
  • Sen, S., & Cohen, A. S. (2020). Sample size requirements for application diagnostic classification models. Frontiers in Psychology, 11,...
  • Sessoms, J., & Henson, R. A. (2018). Applications of diagnostic classification models: A literature review and critical commentary. Measurement:...
  • Sorrel, M. A., Abad, F. J., & Nájera, P. (2021). Improving accuracy and usage by correctly selecting: The effects of model selection in...
  • Sorrel, M. A., Abad, F. J., Olea, J., de la Torre, J., & Barrada, J. R. (2017). Inferential item-fit evaluation in cognitive diagnosis...
  • Sorrel, M. A., Olea, J., Abad, F. J., de la Torre, J., Aguado, D., & Lievens, F. (2016). Validity and reliability of situational judgement...
  • Tatsuoka, K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement,...
  • Templin, J., & Bradshaw, L. (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification,...
  • Testa, S., Toscano, A., & Rosato, R. (2018). Distractor efficiency in an item pool for a statistics classroom exam: Assessing its relation...
  • Tjoe, H., & de la Torre, J. (2014). The identification and validation process of proportional reasoning attributes: An application of...
  • Vermunt, J. K., & Magidson, J. (2013). Latent GOLD 5.0 upgrade manual. Statistical Innovations Inc.
  • Von Davier, M., & Lee, Y. S. (Eds.). (2019). Handbook of diagnostic classification models. Springer.
  • Wang, W., Song, L., Chen, P., Meng, Y., & Ding, S. (2015). Attribute-level and pattern-llevel classification consistency and accuracy...
  • Wu, H. M. (2019). Online individualised tutor for improving mathematics learning: A cognitive diagnosis model approach. Educational Psychology,...
  • Xu, X., & von Davier, M. (2006). Cognitive diagnosis for NAEP proficiency data (Research report ETS RR-06-08). ETS Research Report Series.
  • Yu, X., Cheng, Y., & Chang, H.-H. (2019). Recent developments in cognitive diagnostic computerized adaptive testing (CD-CAT): A comprehensive...
  • Zhan, P. (2020). Longitudinal learning diagnosis: Minireview and future research directions. Frontiers in Psychology, 11, Article 1185. https://...