Xuelan's readings and review

Posterior Predictive Model Checking for Conjunctive Multidimensionality in Item Response Theory

Posterior Predictive Model Checking for Conjunctive Multidimensionality in Item Response Theory

by QIU Xuelan -
Number of replies: 3

Posterior Predictive Model Checking for Conjunctive Multidimensionality in Item Response Theory

Review

Within the framework of item response theory, the key conditional independence assumption of unidimensionality often does not hold. One typical situation is when data exhibit multidimensionality.

Posterior predictive model checking (PPMC) has been applied to assess the model-data fit in Bayesian psychometric modeling. PPMC analyzed characteristics of the observed data and the discrepancy between the observed data and the replicate data which are generated from the posterior predictive distribution. The parameters for the posterior predictive distribution come from the estimate of observed data. If the PPP value (ie, the tail area of the posterior predictive distribution of the discrepancy measure compared to the observed value of the discrepancy measure) is not near zero, it imply the model-data misfit.

The research applied PPMC to the data that follow conjunctive multidimensional models, following the previous research which used the PPMC to compensatory multidimensional IRT models. Similarly, the factors that were considered to be important when detecting the multidimensional were examined: (1) the strength of dependence on auxiliary dimensions in conjunctive MIRT models, (2) sample size; (3) correlations between dimensions, (4) proportion of multidimensional items.

The data in the simulation were generated from a conjunctive multidimensional model and were fit to a unidimensional model. A number of the discrepancy measure which include X2, G2, COV (covariance), MBC, Q3, RESIDCOV (residual item covariance), LOR (log odds ration), standard log odds ration residual), and MH statistics were compared with respect to the effectiveness and sensitivity to the violation of local independence and unidimensionality.

The research found the similar findings of earlier work on situations of compensatory multidimensionality. That is, MBC, Q3, MH statistic is superior to other discrepancy measures.

The paper was well written and easy to follow. We may extend the research to conduct the PPMC to (1) the within and between multidimensional IRT model, (2) to incorporate the polytomous response.

In reply to QIU Xuelan

Re: Posterior Predictive Checking Topic

by LIU CHEN WEI -
I just took a look at this paper one time. Some questions may be unwise.

I found it provided many model-data fit index and compare them in simulation study. If PPMC tells us that the model is fit to data but other index not, how do we make a decision? Have any available package or program up to now? I think it may be some hard for us to check the correctness of program if we write it on our own. Any good idea to be sure of its correctness? I think it can be applied in unfolding model as well when MCMC is used.
In reply to LIU CHEN WEI

Re: Posterior Predictive Checking Topic

by QIU Xuelan -

Q: If PPMC tells us that the model is fit to data but other index not, how do we make a decision?

A: Well, model-data misfit certainly include many kind of violations, multidimensional, or local dependence, or data gen from 3PL but fit with Rasch. Too many situations.

The index in PPMC is just sensitive for certain kind of violation. For example, LR may be sensitive for the local independence but not sensitive for multidimensionality. But Q3 may be sensitive for both kind of violations. Therefore, not any index in PPMC is perfact. We should decide which kind of violation and select proper index in PPMC.

Q: Have any available package or program up to now? I think it may be some hard for us to check the correctness of program if we write it on our own.

A: The most hard part of the program is to compute the index. The replication data could be gen by writing a line of code direclty in WINBUGS.