This study compared three reliability coefficients in a population level. First, two composite score reliability coefficients rho and omega (as well as Cronbach's alpha) were introduced and the relationship among them was stated. The rho which comes from maximum likelihood factor score is demonstrated mathematically always more than omega if essential tau-equivalency assumption do not hold. The the reliability coefficient pi was introduced in the IRT framework. Then the concept of information and reliability was introduced and the relationship of three reliability coefficients were compared mathematically. However, due to the harmfulness of pi and omega come from different reasons of information lose, the magnitude of pi and omega still need to figure out by a numerical analysis approach. Thus simulations were conducted to give the result of comparison of rho, omega and pi. The results show that in typical settings, rho > omega > pi(in this case, pi for dichotomized situation). Further, the factor of discretization in pi was investigated by studied different number of categories and the result show that the pi would greater than omega when the categories more than 10 and close to rho when even more categories. An uncommon scenario was also studied to investigate the relationship between pi and omega. Finally, the relations of three reliability coefficients was summarized and further study directions were mentioned.
This study used the 2PL item response model to do the comparison which involving discrimination parameter, the source of information lose of omega . Whereas the discrimination were not account in the Rasch model, in this case, the pi would just related to the discretization in the words of the authors. Will it different when comparing to the omega?
Another factor in practices can cause the injured reliability is the alignment between person ability and item difficulties (similar to the authors' state: information is a conditional concept which is a function of person ability). By saying so, should the mismatch between population and the item difficulties be investigated? This is also a practical issue when the questionnaires or inventories were administrated to students in order to censor their mental health. However, most of the students were not showing the mental problems. In this case, the questionnaire or inventories will yield very low reliability but not due to the reasons stated in this paper.