15 Comparision of reliability under FA and IRT (Present by Jensen)

Wayne's comment

Wayne's comment

by CHEN Chia Wen -
Number of replies: 0
 

 

When the response is continuous, this study introduce the single factor model and its reliability, the coefficient Omega. We can see this reliability uses unweighted composite score variance as the summation of lamda. This reliability neglects the response pattern information. Relatively, The coefficient Rho uses the maximum likelihood factor score, it is not loss the information about response pattern. In this study the rho is the maximal reliability standard. Then IRT reliability, pi, were introduced and represent the reliability of discrete response data. In mathematically, it is absolutely that the pi smaller than rho, the demonstration was derived as Appendix. it is known that Rho larger than Omega and Pi, how about comparing between Omega and Pi is what this study investigated.

 

According to this simulation study, Omega is almost larger than Pi. That means discrete data would loss more information than unweighted aggregative score, but it is not always. When items load are very differential (item discrimination parameters vary considerably) and the number of categories is larger than five, the Pi will be surpass Omega. Note that this effect is not linear at the number of categories, so it is not worth to add more categories for increasing reliability.

 

 

comment

 

1) This result corresponds with intuitive sense. However, the discrete response is convenience to score and answer, the continuous data relatively are rare in fact. If we can obtain the continuous data, we won't dichotomize it to loss information. If we have only dichotomous data, we will not be able to transform it to continuous data. I think the comparison between these two kind of data is not fair!

 

 

2) In Rasch model, discrimination=1 and lamda(item load)=sigma_error=0.707 are derived. In this case, Cronbach's alpha should be equal to Omega, and I guess Pi will be still smaller then Omega.