El enfoque meta-analítico de generalización de la fiabilidad

• Autores: Julio Sánchez Meca, José Antonio López Pina
• Localización: Acción psicológica, ISSN 1578-908X, Vol. 5, Nº. 2, 2008 (Ejemplar dedicado a: Innovaciones metodológicas en la evaluación psicológica : perspectivas de futuro), págs. 37-64
• Idioma: español
• DOI: 10.5944/ap.5.2.457
• Títulos paralelos:
  • The meta-analytic approach of reliability generalization
• Enlaces
  • Texto completo
• Dialnet Métricas: 18 Citas
• Resumen
  • español

    Frases del tipo «la fiabilidad del test es 0.80» son incorrectas. Es más apropiado decir «la fiabilidad de las puntuaciones del test en una determinada aplicación del mismo es 0.80». El enfoque meta-analítico de generalización de la fiabilidad pretende demostrar que la fiabilidad es una propiedad empírica que varía de una aplicación a otra del test. Este nuevo enfoque meta-analítico está contribuyendo a concienciar a los investigadores sobre la importancia de aportar estimaciones de la fiabilidad con los propios datos y evitar inducciones de la fiabilidad. Se presentan las fases en las que se lleva a cabo un estudio de generalización de la fiabilidad: (a) formulación del problema, (b) búsqueda de los estudios, (c) codificación de los estudios, (d) análisis estadístico e interpretación y (e) publicación. Se presenta una visión actualizada de los problemas estadísticos de este enfoque: (a) transformar versus no transformar los coeficientes, (b) ponderar versus no ponderar los coeficientes, (c) cómo tratar la dependencia estadística entre los coeficientes y (d) cuál es el modelo estadístico más apropiado (efectos fijos, efectos aleatorios, efectos mixtos).

  • English

    Sentences such as «the test reliability is 0.80» are wrong. It is more appropriate to say «the test scores reliability in a given application of it is 0.80». The meta-analytic approach of reliability generalization pretends to show that reliability is an empirical property that varies from one test application to another. This recent meta-analytic approach is helping to make the researchers aware of the importance of reporting reliability estimates obtained from the own data and, of avoiding the malpractice of inducting reliability coefficients from other studies and previous applications of the test. The stages to carry out a reliability generalization study are presented: (a) formulating the problem, (b) searching for the studies, (c) coding studies, (d) statistical analysis and interpretation, and (e) publication. An updated overview of the statistical problems of this approach: (a) to transform versus not to transform the reliability coefficients, (b) to weight versus not to weight the coefficients, (c) how to manage statistical dependency among the coefficients, and (d) which statistical model is the most appropriate (fixed-, random-, and mixed-effects).

• Referencias bibliográficas
  • Barnes, L. L. B., Harp, D. y Jung, W. S. (2002). Reliability generalization of scores on the Spielberger State-Trait Anxiety Inventory. Educational...
  • Beretvas, S. N., Meyers, J. L. y Leite, W. L. (2002). A reliability generalization study of the Marlowe Crowne Social Desirability Scale....
  • Beretvas, S. N. y Pastor, D. A. (2003). Using mixedeffects models in reliability generalization studies. Educational and Psychological Measurement,...
  • Beretvas, S. N., Suizzo, M.-A., Durham, J. A. y Yarnell, L. M. (2008). A reliability generalization study of scores on Rotter’s and Nowicki-Strickland’s...
  • Borenstein, M. J., Hedges, L. V., Higgins, J. C., & Rothstein, H. (2005). Comprehensive meta-analysis: A computer program for research...
  • Botella, J. y Gambara, H. (2002). ¿Qué es el metaanálisis? Madrid: Biblioteca Nueva.
  • Botella, J. y Gambara, H. (2006). Doing and reporting a meta-analysis. International Journal of Clinical and Health Psychology, 6, 425-440.
  • Campbell, J. S., Pulos, S., Hogan, M. y Murry, F. (2005). Reliability generalization of the Psychopathy Checklist applied in youthful samples....
  • Capraro, M. M., Capraro, R. M. y Henson, R. K. (2001). Measurement error of scores on the Mathematics Anxiety Rating Scale across studies....
  • Caruso, J. C. (2000). Reliability generalization of the NEO personality scales. Educational and Psychological Measurement, 60, 236-254.
  • Conway, J. M., Jako, R. A. y Goodman, D. F. (1995). A meta-analysis of interrater and internal consistency reliability of selection interviews....
  • Cooper, H. M. (1998). Integrating research: A guide for literature reviews (2ª ed.). Thousand Oaks, CA: Sage.
  • Cooper, H. y Hedges, L. V. (Eds.) (1994). The handbook of research synthesis. Nueva York: Russell Sage Foundation.
  • Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98-104.
  • Crocker, L. y Algina, J. (1986). Introduction to classical and modern test theory. Nueva York: Holt, Rinehart, & Winston.
  • Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 15, 297- 334.
  • Dawis, R. V. (1987). Scale construction. Journal of Counseling Psychology, 34, 481-489.
  • DerSimonian, R. y Laird, N. (1986). Meta-analysis of clinical trials. Controlled Clinical Trials, 7, 177- 188.
  • Dimitrov, D. M. (2002). Reliability: Arguments for multiple perspectives and potential problems with generalization across studies. Educational...
  • Feldt, L. S. y Brennan, R. L. (1989). Reliability. En R. L. Linn (Ed.), Educational measurement (3ª ed., pp. 105-146). Nueva York: American...
  • Feldt, L. S. y Charter, R. A. (2006). Averaging internal consistency reliability coefficients. Educational and Psychological Measurement,...
  • Glass, G. V., McGaw, B. y Smith, M. L. (1981). Metaanalysis in social research. Beverly Hills, CA: Sage.
  • Gronlund, N. E. y Linn, R. L. (1990). Measurement and evaluation in teaching (6ª ed.). Nueva York: Macmillan.
  • Gulliksen, H. (1987). Theory of mental tests. Nueva York: Wiley.
  • Hakstian, A. R. y Whalen, T. E. (1976). A k-sample significance test for independent alpha coefficients. Psychometrika, 41, 219-231.
  • Hall, S. M. y Brannick, M. T. (2002). Comparison of two random-effects methods of meta-analysis. Journal of Applied Psychology, 87, 377-389.
  • Harwell, M. (1997). An empirical study of Hedges’s homogeneity test. Psychological Methods, 2, 219- 231.
  • Hedges, L. V. (1994). Fixed effects models. En H. Cooper y L. V. Hedges (Eds.), The handbook of research synthesis (pp. 285-299). Nueva York:...
  • Hedges, L. V. y Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
  • Hedges, L. V. y Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486-504.
  • Heldref Foundation (1997). Guidelines for contributors. Journal of Experimental Education, 65, 95- 96.
  • Henson, R. K. (2001). Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement and...
  • Henson, R. K. y Thompson, B. (2002). Characterizing measurement error in scores across studies: Some recommendations for conducting «reliability...
  • Higgins, J. P. T. y Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539-1558.
  • Howell, R. T. y Shields, A. L. (2008). The file drawer problem in reliability generalization: A strategy to compute a fail-safe N with reliability...
  • Huedo-Medina, T., Sánchez-Meca, J., Marín-Martínez, F. y Botella, J. (2006). Assessing heterogeneity in meta-analysis: Q statistic or I2 index?...
  • Hunter, J. E. y Schmidt, F. S. (1990). Methods of metaanalysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.
  • Hunter, J. E. y Schmidt, F. S. (2004). Methods of metaanalysis: Correcting error and bias in research findings (2ª ed.). Thousand Oaks, CA:...
  • Hunter, J. E., Schmidt, F. S. y Jackson, G. B. (1982). Meta-analysis: Cumulating research findings across studies. Beverly Hills, CA: Sage.
  • Leach, L. F., Henson, R. K., Odom, L. R. y Cagle, L. S. (2006). A reliability generalization study of the Self-Description Questionnaire....
  • Li, A. y Bagger, J. (2007). The Balanced Inventory of Desirable Responding (BIDR): A reliability generalization study. Educational and Psychological...
  • Lipsey, M. W. y Wilson, D. B. (2001). Practical metaanalysis. Thousand Oaks, CA: Sage.
  • López-Pina, J. A., Sánchez-Meca, J. y Rosa-Alcázar, A. I. (2008). The Hamilton Rating Scale for Depression: A reliability generalization study....
  • Marín-Martínez, F., Sánchez-Meca, J., Huedo, T. y Fernández-Guzmán, I. (2007). Meta-análisis: ¿Dónde estamos y hacia dónde vamos? En A. Borges...
  • Marín-Martínez, F. y Sánchez-Meca, J. (1998). Testing for dichotomous moderators in meta-analysis. Journal of Experimental Education, 67,...
  • Martín, J. L. R., Tobías, A. y Seoane, T. (Coords.) (2006). Revisiones Sistemáticas en las Ciencias de la Vida. Toledo: FISCAM.
  • Mason, C., Allam, R. y Brannick, M. T. (2007). How to meta-analyze coefficient-of-stability estimates: Some recommendations based on Monte...
  • Miller, M. B. (1995). Coefficient alpha: A basic introduction from the perspectives of classical test theory and structural equation modeling....
  • Onwuegbuzie, A. J. y Daniel, L. G. (2004). Reliability generalization: The importance of considering sample specificity, confidence intervals,...
  • Parker, K. (1983). A meta-analysis of the reliability and validity of the Rorschach. Journal of Personality Assessment, 47, 227-231.
  • Parker, K. C. H., Hanson, R. K. y Hunsley, J. (1988). MMPI, Rorschach, and WAIS: A meta-analytic comparison of reliability, stability, and...
  • Pedhazur, E. J. y Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Erlbaum.
  • Petrosino, A., Boruch, R. F., Soydan, H., Duggan, L. y Sánchez-Meca, J. (2001). Meeting the challenges of evidence-based policy: The Campbell...
  • Petticrew, M. y Roberts, H. (2006). Systematic Reviews in the Social Sciences: A practical guide. Malden, MA: Blackwell.
  • Raykov, T. (1997). Estimation of composite reliability for congeneric measures. Applied Psychological Measurement, 21, 173-184.
  • Rodríguez, M. C. y Maeda, Y. (2006). Meta-analysis of coefficient alpha. Psychological Methods, 11, 306- 322.
  • Rosenberg, M. S., Adams, D. C. y Gurevitch, J. (1999). MetaWin: Statistical software for meta-analysis with resampling tests (Vers. 2.0)....
  • Rosenthal, R. (1991). Meta-analytic procedures for social research (2ª ed.). Newbury Park, CA: Sage.
  • Rosenthal, R. (1995). Writing meta-analytic reviews. Psychological Bulletin, 118, 183-192.
  • Rowley, G. L. (1976). The reliability of observational measures. American Educational Research Journal, 13, 51-59.
  • Salgado, J. F. y Moscoso, S. (1996). Meta-analysis of interrater reliability of job performance ratings in validity studies of personnel selection....
  • Sánchez-Meca, J. (1999). Meta-análisis para la inves- tigación científica. En F. J. Sarabia (Coord.), Metodología para la investigación en...
  • Sánchez-Meca, J. (2003). La revisión del estado de la cuestión: El meta-análisis. En C. Camisón, M. J. Oltra y M. L. Flor (Eds.), Enfoques,...
  • Sánchez-Meca, J. y Ato, M. (1989). Meta-análisis: Una alternativa metodológica a las revisiones tradi- cionales de la investigación. En J....
  • Sánchez-Meca, J., Boruch, R. F., Petrosino, A. y RosaAlcázar, A. I. (2002). La Colaboración Campbell y la práctica basada en la evidencia....
  • Sánchez-Meca, J. y Marín-Martínez, F. (1997). Homogeneity tests in meta-analysis: A Monte Carlo comparison of statistical power and Type I...
  • Sánchez-Meca, J. y Marín-Martínez, F. (1998). Testing continuous moderators in meta-analysis: A comparison of procedures. British Journal...
  • Sánchez-Meca, J. y Marín-Martínez, F. (2008). Confidence intervals for the overall effect size in random-effects meta-analysis. Psychological...
  • Sánchez-Meca, J., Marín-Martínez, F. y Huedo, T. (2006). Modelo de efectos fijos versus modelo de efectos aleatorios. En J. L. R. Martín,...
  • Sawilowsky, S. S. (2000a). Psychometrics versus datametrics: Comment on Vacha-Haase’s ‘Reliability generalization’ method and some EPM editorial...
  • Sawilowsky, S. S. (2000b). Reliability: Rejoinder to Thompson and Vacha-Haase. Educational and Psychological Measurement, 60, 196-200.
  • Schmidt, F. S. y Hunter, J. S. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology,...
  • Schulze, R. (2004). Meta-analysis: A comparison of approaches. Hogrefe & Huber Pub.
  • Shadish, W. R., Chacón-Moscoso, S. y Sánchez-Meca, J. (2005). Evidence-based decision making: Enhancing systematic reviews of program evaluation...
  • Silver , N. y Dunlap, W. (1987). Averaging coefficients: Should Fisher’s z-transformation be used? Journal of Applied Psychology, 72, 3-9.
  • Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837- 847.
  • Thompson, B. (Ed.) (2003). Score reliability: Contemporary thinking on reliability issues. Thousand Oaks, CA: Sage.
  • Thompson, B. y Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement,...
  • Traub, R. E. (1994). Reliability for the social sciences: Theory and applications (Vol. 3). Thousand Oaks, CA: Sage.
  • Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational...
  • Vacha-Haase, T., Henson, R. K. y Caruso, J. C. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability....
  • Vacha-Haase, T., Kogan, L. R. y Thompson, B. (2000). Sample compositions and variabilities in published studies versus those in test manuals....
  • Vacha-Haase, T. y Ness, C. (1999). Practices regarding reporting of reliability coefficients: A review of three journals. Journal of Experimental...
  • Viechtbauer, W. (2005). Bias and efficiency of meta- analytic variance estimators in the randomeffects model. Journal of Educational and Behavioral...
  • Viechtbauer, W. (2007). Confidence intervals for the amount of heterogeneity in meta-analysis. Statistics in Medicine, 26, 37-52.
  • Viswesvaran, C. y Ones, D. S. (2000). Measurement error in ‘big five factors’ personality assessment: Reliability generalization across studies...
  • Whittington, D. (1998). How well do researchers re- port their measures? An evaluation of measurement in published educational research. Educational...
  • Wilkinson, L. & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journal: Guidelines and explanations....
  • Yarnold, P. R. y Mueser, K. T. (1989). Meta-analyses of the reliability of Type A behaviour measures. British Journal of Medical Psychology,...
  • Yin, P. y Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies. Educational...