Detecting Violations of Factorial Invariance Using Data-Based Specification Searches: A Monte Carlo Study
By Yoon, M. & Millsap, R. E.
When testing factorial invariance in multiple-group confirmatory factor analysis (MGCFA), a reference indicator (RI) is often used for model identification. The factor loading for this RI is usually fixed to be 1.0 in all groups. This automatically implies weak (metric) invariance of this indicator, which can lead to biased estimates of other model parameters and hence misleading results when comparing factor loadings across groups. Therefore, choosing a truly invariant RI is crucial in testing factorial invariance.
This article suggests an alternative method which avoids choosing an invariant variable as RI. In their method, the variance of the factor is fixed to 1 in one group but not in the other group; all loadings are constrained to invariance across groups at first. Then the fit indices are checked to see if this metric invariance holds. If not, constraints will be relaxed based on the modification index (MI) sequentially, until the model reaches an adequate fit.
A simulation was carried out to examine whether the partial invariance of the factor loadings across groups can be accurately detected. Results showed that their method performed well as long as the proportion of the non-invariant indicators is small.
However, their study only considered the single factor case. Extending their approach to multiple-factor case is not straightforward if cross-loadings exist. Besides, their method based on MI is post-hoc and has been disputed and discouraged due to its data-driven nature. In fact, data-driven methods have been shown to be very inconsistent over repeated samples, indicating poor generalizability of the resulting model.
Following this article, several directions can be explored:
1) Behavior of their method with ordinal data. In order to adopt this method to categorical data, an item factor model would be the most straightforward model to use. Comparing to a common factor model, it involves new parameters (e.g., threshold parameters), and often requires larger sample size for an adequate efficiency. Therefore, it can be expected that the detection of invariance might be less successful with ordinal data than with continuous data.
2) Behavior of their method on other factorial invariance. This article only examined the (partial) weak invariance, where the factor loadings are constrained to be equal across groups. Tests of intercept invariance (i.e., strong invariance) and unique variance invariance (strong invariance) can be examined as well. Either partial or complete invariance can be tested.
3) Extension of their method to multi-factor / mean structure model. Extension to mean structure model is relatively easy. However, extension to more than one factor might raise additional questions especially when cross-loadings exist. This is because estimates of a variable’s loadings on different factors will interact with each other, such that their contributions on the MI confound. It is difficult to decide which loading(s) should be released to vary across groups according to significant drop of MI.
4) Extension of their method to more than two groups. To do so, several additional steps should be added to explore the partial invariance among three groups. For example, one can 1) test invariance across all groups at first, 2) release invariance constrain in every one of the groups to see which group leads to the largest drop of MI, 3) release invariance constrain in two, three, …, groups sequentially until no significant MI is observed.