Analyzing Two Automatic Latent Semantic Analysis (LSA) Assessment Methods (Inbuilt Rubric vs. Golden Summary) in Summaries Extracted from Expository Texts

• José Ángel Martínez-Huertas ^[1] ; Olga Jastrzebska ^[1] ; Adrián Mencu ^[1] ; Jessica Moraleda ^[1] ; Ricardo Olmos ^[1] ; José Antonio León ^[1]
  1. [1] Universidad Autónoma de Madrid

     Universidad Autónoma de Madrid

     Madrid, España

     
• Localización: Psicología educativa, ISSN 1135-755X, Vol. 24, Nº. 2, 2018, págs. 85-92
• Idioma: inglés
• DOI: 10.5093/psed2048a9
• Títulos paralelos:
  • Análisis de dos métodos de evaluación automática de análisis semántico latente (LSA): un nuevo método LSA (Inbuilt Rubric) y un método LSA tradicional (Golden Sum
• Enlaces
  • Texto completo
• Dialnet Métricas: 2 Citas
• Resumen
  • español

    El objetivo de este estudio es comparar dos métodos de evaluación automática del análisis semántico latente (LSA): Un nuevo método LSA (Inbuilt Rubric) y un método LSA tradicional (Golden Summary). Se analizaron dos condiciones del método Inbuilt Rubric: el número de descriptores léxicos que se utilizan para generar la rúbrica (pocos vs. muchos) y una corrección que penaliza el contenido irrelevante incluido en los resúmenes de los estudiantes (corregido vs. no corregido). Ciento sesenta y seis estudiantes divididos en dos muestras (81 estudiantes universitarios y 85 estudiantes de instituto) participaron en este estudio. Los estudiantes resumieron dos textos expositivos que tenían distinta complejidad (difícil/fácil) y longitud (1,300/500 palabras). Los resultados mostraron que el método Inbuilt Rubric imita las evaluaciones humanas mejor que Golden Summary en todos los casos. La similitud con las evaluaciones humanas fue más alta con Inbuilt Rubric (r = .78 and r = .79) que con Golden Summary (r = .67 and r = .47) en ambos textos. Además, la versión de Inbuilt Rubric con menor número de descriptores y con corrección es la que obtuvo mejores resultados.

  • English

    The purpose of this study was to compare two automatic assessment methods using Latent Semantic Analysis (LSA): a novel LSA assessment method (Inbuilt Rubric) and a traditional LSA method (Golden Summary). Two conditions were d using the Inbuilt Rubric method: the number of lexical descriptors needed to better accommodate an expert rubric (few vs. many) and a weighting to penalize off-topic contents d in the student summaries (weighted vs. non-weighted). One hundred and sixty-six students divided in two different samples (81 undergraduates and 85 High School students) took part in this study. Students summarized two expository texts that differed in complexity (complex/ easy) and length (1,300/500 words). Results showed that the Inbuilt Rubric method simulates human assessment better than Golden summaries in all cases. The similarity with human assessment was higher for Inbuilt Rubric (r = .78 and r = .79) than for Golden Summary (r = .67 and r = .47) in both texts. Moreover, to accommodate an expert rubric into the Inbuilt Rubric method was better using few descriptors and the weighted .

• Referencias bibliográficas
  • Aryal, A., Gallivan, M., & Tao, Y. (2015, August). Using Latent Semantic Analysis to Identify Themes in IS Healthcare Research. Paper...
  • Asimov, I. (1969). Great Ideas of Science. Boston, MA: Houghton Mifflin. Borisov A., Serdyukov P., & de Rijke, M. (2016, April). Using...
  • Cohen, T., Blatter, B., & Patel, V. (2008). Simulating Expert Clinical Comprehension: Adapting Latent Semantic Analysis to Accurately...
  • Dornisch, M. M., & McLoughlin, A. S. (2006). Limitations of web-based rubric resources: Addressing the challenges. Practical Assessment,...
  • Foltz, P. W., Gilliam, S., & Kendall, S. (2000). Supporting contentbased feedback in online writing evaluation with LSA. Interacting Learning...
  • Foltz, P. W., Laham, D., & Landauer, T. (1999). Automated Essay Scoring: Applications to Educational Technology. Proceedings of World...
  • Franzke, M., Kinstch, E., Caccamise, D., Johnson, N., & Dooley, S. (2005). Summary street: computer support for comprehension and writing....
  • Graesser, A. C., Wiemer-Hastings, P., Wiemer-Hastings, K., Harter, D., Person, N., & Tutoring Research Group (2000). Using latent semantic...
  • Günther, F., Dudschig, C., & Kaup, B. (2015). Latent Semantic Analysis cosines as a cognitive similarity measure: Evidences from priming...
  • Hambleton, R. K., & Swaminathan, H. (2013). Item response theory: Principles and applications. Berlin/Heidelberg, Germany: Springer Science...
  • Jonassen, D. H., Peck, K. C., & Wilson, B. G. (1999). Learning with technology in the classroom: A constructivist perspective. New York,...
  • Jorge-Botana, G., León, J. A., Olmos, R., & Escudero, I. (2011). The representation of polysemy through vectors: some building blocks...
  • Jorge-Botana, G., Olmos, R., & Barroso, A. (2013, July). Gallito 2.0: a Natural Language Processing tool to support Research on Discourse....
  • Jorge-Botana, G., Olmos, R., León J. A. (2009). Using LSA and the predication algorithm to improve extraction of meanings from a diagnostic...
  • Kintsch, E., Steinhart, D., Stahl, G., LSA Research Group, Matthews, C., & Lamb, R. (2000). Developing summarization skills through the...
  • Klein, R., Kyrilov, A., & Tokman, M. (2011). Automated assessment of short free-text responses in computer science using latent semantic...
  • Landauer, T. K. & Dumais, S. (1997). A solution to Plato’s problem: The Latent Semantic Analysis theory of the acquisition, induction,...
  • Landauer, T. K., Foltz, P. W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse processes, 25, 259-284. https://doi....
  • Landauer, T. K., McNamara, D. S., Dennis, S., & Kintsch, W. (2007). The Handbook of Latent Semantic Analysis. Mahwah, NJ: Routledge.
  • León, J. A., Olmos, R., Escudero, I., Cañas, J. J., & Salmerón, L. (2006). Assessing short summaries with human judgments procedure and...
  • Madrid, R. I., & Cañas J. J. (2011). Using latent semantic analysis to enhance the comprehensibility of hypertext systems. International...
  • Moskal, B. M. (2000). Scoring Rubrics: What, When and How? Practical Assessment, Research & Evaluation, 7(10), 71-81.
  • Nakov, P., Popova, A., & Mateev, P. (2001, September). Weight functions impact on LSA performance. Presented at the EuroConference Recent...
  • Nitko, A. J., (2004). The educational assessment of students (4th ed.). Englewood Cliffs, NJ: Prentice Hall.
  • Olmos, R., Jorge-Botana, G., León, J. A., & Escudero, I. (2014). Transforming Selected Concepts Into Dimensions in Latent Semantic Analysis....
  • Olmos, R., Jorge-Botana, G., Luzón, J. M., Cordero, J., & León, J. A. (2016). Transforming LSA space dimensions into a rubric for an automatic...
  • O’Reilly, R. C., & Munakata, Y. (2000). Computational Explorations in Cognitive Neuroscience. Cambridge, MA: MIT Press.
  • Pakhomov, S. V., & Hemmy, L. S. (2014). A computational linguistic measure of clustering behavior on semantic verbal fluency task predicts...
  • Peiro, A. (1972). Ciencias de la Naturaleza 6º EGB. Madrid, España: Anaya. Pu, X., Jin, R., Wu, G., Han, D., & Xue, G. R. (2015). Topic...
  • space with keywords. Presented at the 24th ACM International Conference on Information and Knowledge Management (pp. 11411150). Melbourne,...
  • Raykov, T., & Marcoulides, G. A. (2008). An introduction to applied multivariate analysis. New York, NY: Routledge.
  • Rehder, B., Schreiner, M. E., Wolfe, M. B., Laham, D., Landauer, T. K., & Kintsch, W. (1998). Using Latent Semantic Analysis to assess...
  • technical considerations. Discourse Processes, 25, 337-354. https:// doi.org/10.1080/01638539809545031
  • Ryan, J. O., Kaltman, E., Mateas, M., & Wardrip-Fruin, N. (2015). What we talk about when we talk about games: Bottom-up game studies...
  • Tierney, R., & Simon, M. (2004). What’s still wrong with rubrics: focusing on the consistency of performance criteria across scale levels....
  • Visinescu, L., & Evangelopoulos, N. (2014). Orthogonal Rotations in Latent Semantic Analysis: An Empirical Study. Decision Support Systems,...
  • Wendy, S. T. W., How, B. C., & Atoum, I. (2014). Using Latent Semantic Analysis to Identify Quality in Use (QU) Indicators from User Reviews....
  • Wolfe, M. B. W. (2005). Memory for narrative and expository text: Independent influences of semantic associations and text organization. Journal...
  • Xu, H., Zeng, W., Gui, J., Qu, P., Zhu, X., & Wang, L. (2015). Exploring similarity between academic paper and patent based on Latent...