Differential Item Functioning: Beyond validity evidence based on internal structure

• Juana Gómez-Benito ^[1] ; Stephen Sireci ^[2] ; José-Luis Padilla ^[3] ; M. Dolores Hidalgo ^[4] ; Isabel Benítez ^[5]
  1. [1] Universitat de Barcelona

     Universitat de Barcelona

     Barcelona, España

     
  2. [2] University of Massachusetts Amherst

     University of Massachusetts Amherst

     Town of Amherst, Estados Unidos

     
  3. [3] Universidad de Granada

     Universidad de Granada

     Granada, España

     
  4. [4] Universidad de Murcia

     Universidad de Murcia

     Murcia, España

     
  5. [5] Universidad Loyola

     Universidad Loyola

     Bolivia

     

  Mostrar afiliaciones +
• Localización: Psicothema, ISSN-e 1886-144X, ISSN 0214-9915, Vol. 30, Nº. 1, 2018, págs. 104-109
• Idioma: inglés
• Títulos paralelos:
  • Funcionamiento Diferencial del Item: más allá de las evidencias de validez basadas en la estructura interna
• Enlaces
  • Texto completo (pdf)
• Dialnet Métricas: 13 Citas
• Resumen
  • español

    Antecedentes: en la última edición de los Standards for Educational and Psychological Testing, el Funcionamiento Diferencial del Item (DIF) es considerado como una evidencia de validez basada en la estructura interna. Sin embargo, no hay indicaciones claras sobre cómo diseñar un estudio de DIF como un estudio de validación. Proponemos relacionar el DIF con todas las fuentes de evidencias de validez y un esquema conceptual para transformar los estudios “típicos” de DIF en estudios de validación. Método: se lleva a cabo una extensa revisión de la literatura y realizan propuestas teóricas y prácticas. Resultados: el artículo aporta argumentos a favor de abordar la detección e interpretación del DIF como estudios de validación y recomendaciones para realizar estudios de validación sobre el DIF.

    Discusión: la combinación de resultados cuantitativos y cualitativos en un marco de investigación mixta, junto con el diseño de los estudios de DIF como estudios de validación, puede mejorar la validez de las interpretaciones de las puntuaciones en los tests.

  • English

    Background: In the latest release of the Standards for Educational and Psychological Testing, Differential Item Functioning (DIF) is considered as validity evidence based on internal structure. However, there are no indications of how to design a DIF study as a validation study. In this paper, we propose relating DIF to all sources of validity evidence, and provide a general conceptual framework for transforming “typical” DIF studies into validation studies. Method: We perform a comprehensive review of the literature and make theoretical and practical proposals. Results:

    The article provides arguments in favour of addressing DIF detection and interpretation as validation studies, and suggestions for conducting DIF validation studies. Discussion: The combination of quantitative and qualitative data within a mixed methods research perspective, along with planning DIF studies as validation studies, can help improve the validity of test score interpretations

• Referencias bibliográficas
  • American Psychological Association, American Educational Research Association & National Council on Measurement in Education (1974). Standards...
  • American Educational Research Association, American Psychological Association & National Council on Measurement in Education (1999). Standards...
  • American Educational Research Association, American Psychological Association & National Council on Measurement in Education (2014). Standards...
  • Angoff, W. H. (1993). Perspectives on Differential Item Functioning Methodology. In P. W. Holland & H. Wainer (Eds.), Differential Item...
  • Angoff, W. H., & Ford, S. F. (1973). Item-race interaction on a test of scholastic aptitude. Journal of Educational Measurement, 10(2),...
  • Benítez, I., & Padilla, J. L. (2014). Analysis of non-equivalent assessments across different linguistic groups using a Mixed Methods...
  • Benítez, I., Padilla, J. L., Hidalgo, M. D., & Sireci, S. (2015). Using mixed methods to interpret Differential Item Functioning. Applied...
  • Camilli, G., & Shepard, L. A. (1994). Methods for Identifying Biased Test Items. United States: SAGE Publications.
  • Cizek, G.J., Rosenberg, S.L., & Koons, H.H. (2008). Sources of validity evidence for educational and psychological tests. Educational...
  • Cleary, T. A., & Hilton, T. L. (1968). An investigation of item bias. Educational and Psychological Measurement, 28(1), 61-75. https://doi.org/10.1177/001316446802800106
  • Cohen, A.S., & Bolt, D.M. (2002). A mixture model analysis of differential item functioning. Paper presented at the Annual Meeting of...
  • Creswell, J. W. (2015). A concise introduction to mixed methods research. Thousand Oaks, CA: Sage Publications.
  • Crocker, L. (1996). Editorial: The great validity debate. Educational Measurement: Issues and Practice, 16(2), 4. https://doi.org/10.1111/j.1745-3992.1997.tb00584.x
  • Embretson, S. (1983). Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93(1), 179-197. https://doi.org/10.1037/0033-2909.93.1.179
  • Evers, A., Muñiz, J., Hagemeister, C., Høstmælingen, A., Lindley, P., Sjöberg, A., & Bartram, D. (2013). Assessing the quality of tests:...
  • Gadermann, A.M., Guhn, M., & Zumbo, B.D. (2011). Investigating the substantive aspect of construct validity for the Satisfaction with...
  • Gómez-Benito, J., Balluerka, N., González, A., Widaman, K. F., & Padilla, J. L. (2017). Detecting differential item functioning in behavioral...
  • Hernández, A., Tomás, I., Ferreres, A., & Lloret, S. (2015). Tercera evaluación de test editados en España [Third evaluation of tests...
  • Hidalgo, M. D., & Gómez-Benito, J. (2010). Education measurement: Differential item functioning. In P. Peterson, E. Baker & B. McGaw...
  • Hidalgo, M. D., Benítez, I., Padilla, J. L., & Gómez-Benito, J. (2017). How polytomous item bias can affect total-group survey score comparisons....
  • Hidalgo, M.D., López-Martínez, M.D., Gómez-Benito, J., & Guilera, G. (2016). A comparison of discriminant logistic regression and Item...
  • Holland, P. W., & Thayer, D. T. (1988). Differential item performance and Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.),...
  • Kane, M.T. (2006). Validation. In B.L. Robert (Ed.), Educational measurement (4th ed., pp. 17-64). Wesport, CT: Praeger
  • Kane, M.T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1-73. https://doi.org/10.1111/jedm.12000
  • Lane, S. (2014). Validity evidence based on testing consequences. Psicothema, 26(1), 127-135. https//doi.org/10.7334/psicothema2013.258
  • Maddox, B., Zumbo, B. D., Tay-Lim, B., & Qu, D. (2015). An anthropologist among the psychometricians: Assessment events, Ethnography,...
  • Mellenbergh, G. J. (1982). Contingency table models for assessing item bias. Journal of Educational Statistics, 7(2), 105-118. https://doi....
  • Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13-103). New York: MacMillan.
  • Penfield, R. D., & Lam, T. C. M. (2000). Assessing differential item functioning in performance assessment: Review and recommendations....
  • Prieto, G., & Muñiz, J. (2000). Un modelo para evaluar la calidad de los tests utilizados en España [A model for evaluating quality of...
  • Rogers, J., & Swaminathan, H. (2016). Concepts and methods in research on differential item functioning of tests items. Past, present,...
  • Samuelson, K. (2005). Examining Differential Item Functioning from a latent class perspective. Dissertation Thesis. University of Maryland.
  • Sireci, S. G. (2005a). Using bilinguals to evaluate the comparability of different language versions of a test. In R.K. Hambleton, P. Merenda...
  • Sireci, S. G. (2005b). Unlabeling the disabled: A perspective on flagging scores from accommodated test administrations. Educational Researcher,...
  • Sireci, S. G. (2016). On the validity of useless tests. Assessment in Education: Principles, Policy & Practice, 23(2), 226-235. https://doi.o...
  • Sireci, S. G., & Padilla, J. L. (2014). Validity assessment: Introduction to the special section. Psicothema, 26(1), 97-99. https//doi.org/10.7334/psicothema2013.255
  • Sireci, S. G., & Ríos, J.A. (2013). Decisions that make a difference in detecting differential item functioning. Educational Research...
  • Spearman, C. (1904).The proof and measurement of association between two things. American Journal of Psychology, 15(1), 72-101. https://doi.org/10.2307/1412159
  • Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational...
  • Zumbo, B.D. (2007). Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessment...
  • Zumbo, B. D., Liu, Y., Wu, A. D., Shear, B. R., Olvera, O. L., & Tanvinder, K. A. (2015). A methodology for Zumbo’s third generation DIF...