This paper examines the factor ratio test and the stepwise partitioning procedure in combination in terms of accuracy of identifying invariant referent variables in multiple-group CFA.
To identify the invariant subsets of variables, factor ratio tests are carried out first. Each variable is used in turn as the referent and test each other variables for invariance. All pairs of variables are tested in the attempt to identify invariant pairs. Once noninvariant pairs are identified, the SP procedure is used to identify the invariant set by sorting into noninvariant pairs iteratively in several steps.
A Monte Carlo simulation study is carried out by applying factor ratio tests with SP procedures. The manipulated factors are sample size, number of factors, number of indicators per factor and percent of noninvariant indicators. No surprising results are presented in their study. The factor ratio test and the stepwise partitioning procedure in combination maintain a reasonable Type I error rate of identifying the invariance variables set. As the number of indicators increases, the Type I error inflates and the power becomes lower; as the percentage of noninvariant variables increases, the power becomes lower as well.
In discussion, the authors stated the potentially problematic issue, standardization, in MCFA, which is a circular situation: (a) the referent variable must be invariant, (b) invariance cannot be established without estimating a model, and (c) model estimation requires an invariant referent, which brings the process back to the original invariant referent assumption. One important point noticed by the authors is that this circular conundrum is parallel to DIF analysis in IRT, and expected that since purification has been shown to clean the anchored items and lead to more accurate DIF detection, a similar procedure with MCFA would be appropriate.
Another valuable argument from the authors is that they do not simply look at the true positive (TP) and false positive (FP) rates, which can be looked as the “power” and “Type I error” of identifying the invariant variable set, as other researchers did on this topic. Prior to examining TP and FP, they establish that the structural models fit the data properly. Results of TP and FP are based on well-fitted models only. This is because that TP and FP based on all models regardless of their fitness might reflect model misfit as well as or instead of identification of noninvariant referent variables. Such a result would make it difficult or even impossible to determine whether a significant result for an invariance test truly reflects group differences on a proposed referent variable or simple model misfit.
Their simulation study has some limitations, not to mention the nontrivial nature (huge numbers of factor ratio tests to conduct and hence the complicated and rather time intensive SP procedure) of the factor ratio tests with the SP procedure when there are many indicators involved:
1. Only balanced design (equal sample size across groups) is considered.
2. No more than 34% percentage of invariant indicators is considered.
3. Only six indicators per factor is considered, and the total number of indicators is at most 12.
4. Only invariance in factor loadings at a fixed magnitude is considered, and this invariance (0.25) is relatively large for their factor loadings (about 0.6). They should consider median and small magnitude of invariance.