This study designed a comprehensive simulation to investigate the performance of various information criteria in selecting correct models when both mean and covariance structure were unknown. The simulation generated data sets under 192 different conditions, with 2 mean structure levels (full, reduced), 2 repeated measures times (4, 8), 2 population distributions (multivariate normal, moderately skewed), 2 estimation methods (REML, ML), 3 sample size levels (30,60,120), and 4 covariance structures (AR, ARH, TOEPH, UN).
Results were similar for criteria under various conditions, except for the performance of estimation methods in full and reduced models. For full models, the REML1 was better than ML and REML2 at selecting the correct models. However, for the reduced models, the REML2 performed equivalent to ML and both better than REML1. Despite this, the overall results implicated: 1) all criteria performed better as sample size and repeated times increased; 2) the consistent criteria tended to select simple models, when the efficient criteria preferred the complex models; 3) consistent criteria based on total number of subjects were more effective than that on total number of observations.; 4) all criteria were capable in selecting correct models when data were from normal distribution, but none of them performed well when data were from moderately skewed distributions.
Q:
1. Why so many covariance structures used as candidates? Some of them may act very similarly to each other, and would make it hard for the criteria to select and produce incorrect decision.
2. As the results showed, none of the criteria behaved correctly under all the conditions, probably combining some of them can provide more precise decision. But another question arises, if there is contradiction between the selections for different criteria, how to make the final decision?
3. None of the criteria performed well when data were not from normal distribution. If the data is proved to be non-normal, what procedure could be used for model selection?