Comparación de Métodos Estadísticos de Meta-Análisis de Generalización de la Fiabilidad con Datos Reales

• Raimundo Aguayo-Estremera ^[1]
  1. [1] Universidad Complutense de Madrid

     Universidad Complutense de Madrid

     Madrid, España

     
• Localización: Revista iberoamericana de diagnóstico y evaluación psicológica, ISSN 1135-3848, ISSN-e 2183-6051, Vol. 2, Nº 72, 2024, págs. 157-168
• Idioma: español
• DOI: 10.21865/RIDEP72.2.11
• Títulos paralelos:
  • A Comparison of Statistical Methods of Reliability Generalization Meta-Analysis with Real Datasets
• Enlaces
  • Texto completo (pdf)
• Resumen
  • español

    La fiabilidad es una propiedad que deben mostrar todos los tests. Dado que no es extrapolable de una muestra a otra, para estudiarla se desarrolló el meta-análisis de generalización de la fiabilidad. Actualmente existen muchos métodos estadísticos, pero no hay acuerdo sobre el más adecuado para este tipo de meta-análisis. Este trabajo compara en tres conjuntos de datos reales 15 métodos estadísticos distintos en relación a la distribución de coeficientes de fiabilidad, la estimación de la fiabilidad promedio y el efecto de las variables moderadoras.

    Los resultados indicaron que las distribuciones tienden a normalizarse con las transformaciones. Las discrepancias en las estimaciones de la fiabilidad promedio fueron pequeñas, pero pueden llevar a distintas conclusiones sustantivas. Las variables moderadoras obtuvieron resultados significativos en función del método empleado. Se debate la forma en que esta discrepancia entre métodos puede repercutir en aplicaciones concretas de estudios de generalización de la fiabilidad.

  • English

    The reliability is a property that all tests must exhibit. Since it is not generalizable from one sample to another, the meta-analysis of reliability generalization was developed. Currently, there are many statistical methods, but there is no agreement on the most suitable for this type of meta-analysis. This study compares 15 different statistical methods in three real datasets regarding the distribution of reliability coefficients, the estimation of average reliability, and the effect of moderator variables. The results indicated that distributions tend to achieved normality with the transformations. Discrepancies in estimates of average reliability were small but could lead to different substantive conclusions. Moderator variables yielded significant results depending on the method employed. The discussion revolves around how this discrepancy between methods may influence on specific applications of reliability generalization studies.

• Referencias bibliográficas
  • Aguayo, R., Vargas, C., de la Fuente, E. I., & Lozano, L. M. (2011). A meta-analytic reliability generalization study of the Maslach Burnout...
  • Beretvas, S. N., Meyers, J. L., & Leite, W. L. (2002). A reliability generalization study of the Marlowe-Crowne Social Desirability Scale....
  • Bonett, D. G. (2002). Sample size requirements for testing and estimating coefficient alpha. Journal of Educational and Behavioral Statistics,...
  • Bonett, D. G. (2008). Meta-analytic interval estimation for bivariate correlations. Psychological Methods, 13, 173-181. https://doi.org/10.1037/a0012868...
  • Bonett, D. G. (2009). Meta-analytic interval estimation for standardized and unstandardized mean differences. Psychological Methods, 14, 225-238....
  • Bonett, D. G. (2010). Varying coefficient meta-analytic methods for alpha reliability. Psychological Methods, 15, 368-385. https://doi.org/10.1037/a0020142...
  • Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2015). Introduction to meta-analysis (2nd ed.). Wiley.
  • Botella, J., & Ponte, G. (2011). Effects of the heterogeneity of the variances on reliability generalization: An example with the Beck...
  • Botella, J., Suero, M., & Gambara, H. (2010). Psychometric inferences from a meta-analysis of reliability and internal consistency coefficients....
  • Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Holt, Rinehart, & Winston.
  • Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.
  • Feldt, L. S., & Charter, R. A. (2006). Averaging internal consistency reliability coefficients. Educational and Psychological Measurement,...
  • Field, A. P. (2005). Is the meta-analysis of correlation coefficients accurate when population correlations vary? Psychological Methods, 10,...
  • Fisher, R. A. (1921). On the probable error of a coefficient of correlation deduced from a small sample. Metron, 1, 3–32.
  • Gronlund, N. E., & Linn, R. L. (1990). Measurement and evaluation in teaching. (6th ed.). Macmillan.
  • Gross, J. (2023). Nortest: Test for normality. R package version 1.0-4. Disponible en https://CRAN.R-project.org/package=nortest
  • Hakstian, A. R., & Whalen, T. E. (1976). A k-sample significance test for independent alpha coefficients. Psychometrika, 41, 219–231....
  • Hartung, J. (1999). An alternative method for meta-analysis. Biometrical Journal, 41, 901-916. https://doi.org/10.1002/(SICI)1521-4036(199912)41:83.0.CO;2-W...
  • Hedges, L. V., & Olkin, I. (1985). Statistical methods in meta-analysis. Academic Press.
  • Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486-504. https://doi:10.1037//1082-989X.3.4.486...
  • Henson, R. K., & Thompson, B. (2002). Characterizing measurement error in scores across studies: Some recommendations for conducting “reliability...
  • Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting errors and bias in research findings (2nd ed.). Sage.
  • Knapp, G., & Hartung, J. (2003). Improved tests for a random effects meta-regression with a single covariate. Statistics in Medicine,...
  • Konstantopoulos, S., & Hedges, L. V. (2009). Analyzing effect sizes: Fixed-effects models. En H. Cooper, L. V. Hedges & J. C. Valentine,...
  • Laird, N. M., & Mosteller, F. (1990). Some statistical methods for combining experimental results. International Journal of Technology...
  • Leach, L. F., Henson, R. K., Odom, L. R., & Cagle, L. S. (2006). A reliability generalization study of the Self-Description Questionnaire....
  • Lloret-Segura, S., Ferreres-Traver, A., Hernández-Baeza, A., & Tomás-Marco, I. (2014). El análisis factorial exploratorio de los ítems:...
  • López-López, J. A., Botella, J., Sánchez-Meca, J., & Marín-Martínez, F. (2013). Alternatives for Mixed-Effects Meta-Regression Models...
  • López-Pina, J. A., Sánchez-Meca, J., & López-López, J. A. (2012). Methods for averaging alpha coefficients in reliability generalization...
  • Maslach, C., & Jackson, S. E. (1981). The measurement of experienced burnout. Journal of Organizational Behavior, 2, 99-113.
  • Mason, C., Allam, R., & Brannick, M. T. (2007). How to meta-analyze coefficient-of-stability estimates: Some recommendations on Monte...
  • Nunnally, J. C., & Bernstein, I. H. (1978). Psychometric testing. McGraw-Hill.
  • Ondé, D., & Alvarado, J. M. (2022). Contribución de los modelos factoriales confirmatorios a la evaluación de estructura interna desde...
  • Paniagua, D., Alvarado, J. M., Olivares, M., Ruiz, I., Romero-Suárez, M., & Aguayo-Estremera, R. (2022). Estudio de Seguimiento de las...
  • Paniagua, D., Sánchez-Iglesias, I., Miguel-Álvaro, A., Casas-Aragonez, N., Aparicio-García, M. A., & Aguayo-Estremera, R. (2022). Prácticas...
  • Raudenbush, S. W. (2009). Analyzing effect sizes: Random-effects models. En H. Cooper, L. V. Hedges & J. C. Valentine, (Eds.). The handbook...
  • Revelle, W. (2023). psych: Procedures for psychological, psychometric, and personality research. Northwestern University, Evanston, Illinois,...
  • Rodriguez, M. C., & Maeda, Y. (2006). Meta-analysis of coefficient alpha. Psychological Methods, 11, 306-322. https://doi.org/10.1037/1082-989X.11.3.306...
  • Romano, J. L., & Kromrey, J. D. (2009). What are the consequences if the assumption of independent observations is violated in reliability...
  • Rouse, S. V. (2007). Using reliability generalization methods to explore measurement error: An illustration using the MMPI-2 PSY-5 scales....
  • Sánchez-Meca, J., & López-Pina, J. A. (2008). El enfoque meta-analítico de generalización de la fiabilidad. Acción Psicológica, 5, 37-64....
  • Sánchez-Meca, J., López-Pina, J. A., & López-López, J. A. (2008). Una revisión de los estudios meta-analíticos de generalización de la...
  • Sánchez-Meca, J., López-López, J. A., & López-Pina, J. A. (2013). Some recommended statistical analytic practices when reliability generalization...
  • Sawilowsky, S. S. (2000). Psychometrics versus datametrics: Comment on Vacha-Haase’s ‘reliability generalization’ method and some EPM editorial...
  • Scherer, R., & Teo, T. (2020). A tutorial on the meta-analytic structural equation modeling of reliability coefficients. Psychological...
  • Schmidt, F. L. (2010). Detecting and correcting the lies that data tell. Perspectives on Psychological Science, 5, 233–242. https://doi.org/10.1177/174569161036933...
  • Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement,...
  • Vacha-Haase T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational...
  • Vacha-Haase, T., & Thompson, B. (2011). Score reliability: A retrospective look back at 12 years of reliability generalization studies....
  • Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36, 1-48. https://doi.org/10.18637/jss.v036.i03