Testing Factorial Invariance in Multilevel Data: A Monte Carlo Study
Eun Sook Kim
University of South Florida
Oi-man Kwok and Myeongsun Yoon
Department of Educational Psychology, Texas A&M University
The study investigates the performance of measurement invariance testing when the nested data structure is ignored and instead a regular single-level model is utilized. Type I error rate and statistical power are examined both.
Two Monte Carlo studies were conducted.
Study 1: the noninvariance was simulated in the organizational units to investigate the between-level factorial invariance testing with a between-level group membership.
Study 2: it focused on testing factorial invariance when the group membership was at the within level.
Conditions:
Number of clusters (CN): 30, 50, 80,160
Cluster size (CS):10, 20
ICC: 0.09, 0.20, 0.33(with-level variance 1, between level: 0.09, 0.20, 0.33)
1000 replications
The results show that for Multilevel CFA, large ICC with large CN has high power. For ordinary CFA, the type I error was inflated, and both power and type I error were strongly associated with CS
Comment:
Since the ordinary multiple CFA do not consider the clusters so it is reasonable that the power and type I error were not associated with CN.
In the real situation, the invariance may happen both in the within level and between level, for example, the factors gender under the within level, and the different schools in the between level, should we combine all these together?
Sometimes the groups are more than two then will the procedure deal with such problems in one turn or should we separate them for example 3 groups, we need do 3 time for measurement invariance.
If the CN is too small, then the power is so low, then how can we judge the results in the real situation is right which we know nothing about which item is non-invariance. Shall we go back to the ordinary MCFA？
what about the accuracy of the parameters？