Comparison of Reliability Measures Under Factor Analysis and Item Response Theory
Ying Cheng1,2, Ke-Hai Yuan1, and Cheng Liu1
Educational and Psychological Measurement 72(1) 52–67
The purpose of this article is to establish the relationship between an IRT-based reliability coefficient
(p) using the information function and two factor-analysis–based reliability measures (w and r).
The relationship is studied using the equivalency between the unidimensional normal ogive IRT models and the unifactor models. The three reliability coefficients were connected using the concept of information.
The simulation and numerical examples were used to demonstrate the relationships.
The results show that p(2) and w are both smaller than , when item parameters are within typical ranges of psychological and educational assessments, p(2) is smaller than w.
w
The single-factor model:
Xj=ljq+ej
Suppose we have m items and let
X=T+e= + j
w is the ratio of the variance of the true score over the variance of the unweighted composite score X
the widely used Cronbach’s alpha is a lower bound of w.
‘‘maximal reliability” r
It considered the fact that people with the same sum score can have completely different response pattern. r is the highest possible reliability that a test can achieve which can also be achieved by finding the optimally weighted composite score.
More specifically, r dominates w and they are equal only when the ratio of factor loading and the unique variance holds constant across all items.
≥
Item Response Theory Models
The equivalence can be established between the two-parameter normal ogive model and the item
factor analysis model by setting the constrains between the parameters in the two models.
Reliability= I / (1+I)
Define the reliability under dichotomous IRT model as p(2):
p(2) = I /[1+ Eq (1+I(2,q))]
After illustrating, the result comes out to that p(2) and w are both smaller than because of information loss. For p(2), it is because of dichotomization of continuous item responses; for w, it is because of ignoring response pattern.
Comments:
The basic thought is from the reliability of CTT in which X=T+E. After the assumption is satisfied,the reliability can be computed through the variance of the observed score and the error.
Since the printed paper is black the color can’t identified for the simulation result.
What about the effect of different item number?
Only the unifactor model is used in this study, what will happen when the model is more complex?