ARC Laboratory Sharing

REVIEW on Estimating Reliability for Multidimensional Composite Scale Scores

Motivation of Reading This Paper
In my own study, I try to do a systematic simulation to investigate the efficiency of employing multidimensional models to multidimensional dataset rather than treat each subscale as unidimensional test. The average relative efficiency (ARE; Wang & Chen, 2004) was employed to evaluate the analysis, as well as Spearman-Brown formula for evaluate the increasing of test length. In this case, reliability of each subscale was involved. But there is no fixed setting on the reliability when generating datasets. So it is hard to claim that which type of analysis recovered the “true” reliability and it raised a potential dangerous of using the ARE and Spearman-Brown formula.

Main Ideas of This Paper
By conducted a series of simulation, this paper compared four methods for reliability of a multidimensional factor-structure datasets. They are Coefficient-alpha, Stratified-alpha, Maximal Reliability, and Multidimensional omega. And the results shows Stratified-alpha recover best and the commonly use Coefficient-alpha recover worst.

Outcomes
The most valuable outcome by reading this paper is the simulation methods of generating a dataset with fixed reliability. However, I am still hard to apply it in the context of item response theory. In the paper, the reliability estimation was investigated under the score matrix, it is straight forward. In IRT simulation study, there is dataset with responses rather observed-score covariance matrix, the gap of how to manipulate the covariance matrix to item response is confusing. Also, what is the relationship between reliability and information in the context of item response theory.

To get deeper understanding of the above two questions, the following paper will be read in the coming days.
http://www.springerlink.com/content/8883234526n345xv/fulltext.pdf