Modeling Local Item Dependence in Cloze and Reading Comprehension Test Items Using Testlet Response Theory

• Purya Baghaei ^[1] ; Hamdollah Ravand ^[2]
  1. [1] English Department, Islamic Azad University, Mashhad Branch, Mashhad, Iran
  2. [2] Vali-e-Asr University of Rafsanjan, Iran, and University of Jiroft, Iran
• Localización: Psicológica: Revista de metodología y psicología experimental, ISSN-e 1576-8597, ISSN 0211-2159, Vol. 37, Nº 1, 2016, págs. 85-104
• Idioma: inglés
• Enlaces
  • Texto completo (pdf)
• Dialnet Métricas: 2 Citas
• Resumen

  • In this study the magnitudes of local dependence generated by cloze test items and reading comprehension items were compared and their impact on parameter estimates and test precision was investigated. An advanced English as a foreign language reading comprehension test containing three reading passages and a cloze test was analyzed with a two-parameter logistic testlet response model and a two-parameter logistic item response model.

    Results showed that the cloze test produced substantially higher magnitudes of local dependence than reading items, albeit the levels of local dependency produced by reading items was not ignorable. Further analyses demonstrated that while even substantial magnitudes of testlet effect does not impact parameter estimates it does influence test reliability and information.

    Implications of the research for foreign language proficiency testing, where testlets are regularly used, are discussed.

• Referencias bibliográficas
  • Alderson, J. C. (1979). The cloze procedure as a measure of proficiency in English as a foreign language. TESOL Quarterly, 13, 219-227. DOI:...
  • Alderson, J. C. (1980). A process approach to reading at the University of Mexico. ELT Documents: Special Issue. Projects in Materials Design,...
  • Alderson, J. (2000). Assessing reading. New York: Cambridge University Press
  • Bachman, L. F. (1985). Performance on cloze tests with fixed-ratio and rational deletions. TESOL Quarterly, 19, 535-556. DOI: 10.2307/3586277.
  • Bachman, L.F. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press.
  • Baghaei, P. & Aryadoust, V. (2015). Modeling local item dependence due to common test format with a multidimensional Rasch model. International...
  • Baghaei, P. (2012). The application of multidimensional Rasch models in large scale assessment and validation: An empirical example. Electronic...
  • Baghaei, P. (2011). Optimal number of gaps in C-Test passages. International Education Studies, 4, 166-171. DOI: 10.5539/ies.v4n1p166.
  • Baghaei, P. (2010). A comparison of three polychotomous Rasch models for super-item analysis. Psychological Test and Assessment Modeling,...
  • Baghaei, P. (2007). Local dependency and Rasch measures. Rasch Measurement Transactions. 21, 1105-1106.
  • Beretvas, S. N., & Walker, C. M. (2012). Distinguishing differential testlet functioning from differential bundle functioning using the...
  • Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical...
  • Bormuth, J. R. (1967). Comparable cloze and multiple-choice comprehension test scores. Journal of Reading, 10, 291-299.
  • Bormuth, J. R. (1968a). Cloze test readability: Criterion reference scores. Journal of Educational Measurement, 5, 189-196. DOI: 10.1111/j.1745-3984.1968.tb00625.x.
  • Bormuth, J. R. (1968b). The cloze readability procedure. Elementary English, 45, 429-436
  • Bormuth, J. R. (1969). Factor validity of cloze tests as measures of reading comprehension ability. Reading Research Quarterly, 4, 358-365....
  • Bradlow, E. T., Wainer, H., & Wang, X., (1999). A Bayesian random effects model for testlets. Psychometrika, 64, 153-168. DOI: 10.1002/j.2333-8504.1998.tb01752.x
  • Chang, Y., & Wang, J. (2010, July). Examining testlet effects on the PIRLS 2006 assessment. Paper presented at the 4th IEA International...
  • Chavez-Oller, M.A., Chihara, T., Weaver, K.A., & Oller, J.W. Jr. (1985). When are cloze items sensitive to constraints across sentences?...
  • Chihara, T., Oller, J. W. Jr., Weaver, K. A. & Chavez-Oller, M. A. (1977). Are cloze items sensitive to constraints across sentences?...
  • Eckes, T., & Baghaei, P. (2015). Using testlet response theory to examine local dependence in C-Tests. Applied Measurement in Education,...
  • Eckes, T. (2014). Examining testlet effects in the TestDaF listening section: A testlet response theory modeling approach. Language Testing,...
  • Farhady, H. (1983). New directions for ESL proficiency testing. In J. W. Oller, Jr. (Ed.), Issues in language testing research (pp. 253-269)....
  • Fox, J. P. (2010). Bayesian item response modeling: Theory and applications. New York: Springer. DOI: 10.1111/j.1751-5823.2011.00159_1.x.
  • Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2004). Bayesian data analysis (2nd Ed.). Boca Raton, FL: Chapman & Hall/CRC.
  • Gilliland, J. (1972). Readability. London: University of London Press for the United Kingdom Reading Association.
  • Glas, C. A. W., Wainer, H., & Bradlow, E. T. (2000). MML and EAP estimation in testletbased adaptive testing. In W. J. van der Linden...
  • Greene, B. (2001). Testing reading comprehension of theoretical discourse with cloze. Journal of Research in Reading, 24, 82-98. DOI: 10.1111/1467-9817.00134.
  • Grotjahn, R. (1987). How to construct and evaluate a C-Test: a discussion of some problems and some statistical analyses. In R. Grotjahn,...
  • Harrison, C. (1980). Readability in the classroom. Cambridge: Cambridge University Press.
  • Ip, E. H. (2010). Interpretation of the three-parameter testlet response model and information function. Applied Psychological Measurement,...
  • Jiao, H., Wang, S., & Kamata, A. (2005). Modeling local item dependence with the hierarchical generalized linear model. Journal of Applied...
  • Jonz, J. (1990). Another turn in the conversation: What does cloze measure? TESOL Quarterly, 24, 61-83. DOI: 10.2307/3586852.
  • Kibby, M. W. (1980). Intersentential processes in reading comprehension. Journal of Literacy Research, 12, 299-312. DOI: 10.1080/10862968009547383.
  • Klare, G. R. (1984). Readability. In P. D. Pearson (Ed.), Handbook of reading research (pp. 681-744). New York: Longman.
  • Klein-Braley, C. (1997). C-Tests in the context of reduced redundancy testing: An appraisal. Language Testing, 14, 47-84. DOI: 10.1177/026553229701400104.
  • Linacre, J. M. (2012). A user’s guide to WINSTEPS-MINISTEP: Rasch-model computer programs. Chicago, IL: Winsteps.com.
  • Markman, P.L. (1985). Rational deletion cloze and global comprehension in German. Language Learning, 35, 423-30. DOI: 10.1111/j.1467-1770.1985.tb01085.x
  • McKenna, M. C., & Layton, K. (1990). Concurrent validity of cloze as a measure of intersentential comprehension. Journal of Educational...
  • Oller, J.W. Jr. (1979). Language tests at school. London: Longman
  • Oller, J.W. Jr. (1972). Scoring methods and difficulty levels for tests of proficiency in English as a second language. Modern Language Journal,...
  • Raatz, U. & Klein-Braley, C. (2002). Introduction to language testing and to C-Tests. In J. Coleman, R. Grotjahn, & U. Raatz, (Eds.)...
  • Rankin, E. F. (1959). The cloze procedure: its validity and utility. In Eighth Yearbook of the National Reading Conference (Vol. 8, pp. 131-144)....
  • Ravand, H. (2015). Assessing testlet effect, impact, differential testlet, and item functioning using cross-classified multilevel measurement...
  • Reckase, M. D. (2009). Multidimensional item response theory. New York: Springer. DOI: 10.1007/978-0-387-89976-3.
  • Rye, J. (1982). Cloze procedure and the teaching of reading. London: Heinemann.
  • Samejima, F. (1969). Estimation of ability using a response pattern of graded scores. Psychometrika Monograph No. 17. Richmond, VA: Psychometric...
  • Schroeders, U., Robitzsch, A., & Schipolowski, S. (2014). A comparison of different psychometric approaches to modeling testlet structures:...
  • Shanahan, T., Kamil, M. L., & Tobin, A. W. (1982). Cloze as a measure of intersentential comprehension. Reading Research Quarterly, 17,...
  • Sigott, G. (2004). Towards identifying the C-Test construct. Frankfurt/am: Peter Lang
  • Thissen, D., Steinberg, L., & Mooney, J. A. (1989). Trace lines for testlets: A use of multiple-categorical-response models. Journal of...
  • Wainer, H. (1995). Precision and differential item functioning on a testlet-based test: The 1991 Law School Admissions Test as an example....
  • Wainer, H., Bradlow, E. T., & Du, Z. (2000). Testlet response theory: An analog for the 3- PL useful in testlet-based adaptive testing....
  • Wainer, H., Bradlow, E.T., & Wang, X. (2007). Testlet response theory and its applications. Cambridge: Cambridge University Press.
  • Wainer, H., & Wang, X. (2000). Using a new statistical model for testlets to score TOEFL. Journal of Educational Measurement, 37, 203-220....
  • Wang, X., Bradlow, E. T., & Wainer, H. (2005).User’s guide for SCORIGHT (Version 3.0): A computer program for scoring tests built of testlets...
  • Wang, W. C. & Wilson, M. (2005). The Rasch testlet model. Applied Psychological Measurement, 29, 126-149. DOI: 10.1177/0146621604271053.
  • Wang, X., Bradlow, E. T., & Wainer, H. (2002). A general Bayesian model for testlets: Theory and applications. Applied Psychological Measurement,...
  • Yamashita, J. (2003). Processes of taking a gap-filling test: comparison of skilled and less skilled EFL readers. Language Testing, 20, 267-293....
  • Yen, W. M. (1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement, 30,...
  • Yen, W. M., & Fitzpatrick, A. R. (2006). Item response theory. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 111-153)....
  • Zhang, B. (2010). Assessing the accuracy and consistency of language proficiency classification under competing measurement models. Language...
  • Zumbo, B. D. (1999). A Handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary...