ARC Laboratory Sharing

Attached is my comment.

The subdimension Rasch model also handles the LID by introducing the testlet parameters. The author argued that the testlet effect should interact with each other, that is, correlations. Because some testlets may be similar to each other and have similar effect on people. Then the model can be regarded as a special case of MRCMLM and estimation is available in software Conquest. The author showed us how to use and write the command line in Conquest step by step. An empirical data were analyzed by several models to see which model is more fit to the data.

Questions:

1. If a new testlet is highly correlated to an existing old testlet, why not combine (or discard) them together to avoid the correlation between testlet effects? It is known that it is hard to estimate two random effects when they are highly correlated and can be seen as the same effect in practice.

2. It seems that model with numerous parameters would more fit to data than simpler model. If the correlation between testlet effects is small, is it worthy of use?

3. If the trait is correlated with testlet effect, how can we assess the correlation? It is not as easy as in the testlet Rasch model to explain it.

Comment

The multi-dimensional model, unidimenisonal model and testlet model were compared with the proposed “subdimension model”. It was argued by the author that, compared with other models, the new proposed model can better model data when 1) the variance of subtests are similar in size, 2) matrix sampling is used, and 3) interpreting subtest in large-scale assessment, especially for items showing deviance from a unidimensional model during item selection process.

In the empirical data analysis, the author argued that the subdimension model helps analyze data from large-scale assessment better than testlet model, unidimensional model and multi-dimensional model do. But the improvements seem little.

Reflections:
• It outperformed the multi-dimensional and testlet models very little. The practicality of the proposed model is questioned.
• What is criterion of choosing a dimension where the constraint is applied?
• If some dimensions are highly correlated, is it better to put them as one?
• The article should provide simulation studies that generate data and then estimated from the proposed model. E.g. By manipulating:
• testlet correlations x
• number of items in a testlet x
• levels of local dependence effect x
• Types of testlet items x
• Number of independent items

Estimation of a Rasch model including subdimensions

This paper proposed the subdimension model to solve the problem of local item dependence (LID) which would arise when test analyst directly use unidimensional model and multidimensional model. Also, the proposed model considers correlation between subdimensions which is an extend model of the Rasch testlet model but is a special case of the multidimensional random coefficients multinomial logit model (MRCMLM). The author used the subdimension model to fit the TIMSS 2003 data by CONQUEST along with the unidimensional model, the multidimensional model, and the Rasch testlet model. Results illustrates that the proposed model has the best fit of -2 Log Likelihood.

Sharing, Question and Future study:

1) According to the TIMSS 2003 data, I’m a bit suspicious of the data format is a number of testlets due to the fact that the subdimensions the author depicted here seems to be different contents only. Even if the test format is testlet, how do we decide to choose which model (of the Rasch testlet model and the subdimension model) to analysis the data? Why I’m asking the question is because that -2 Log Likelihood of these two models in table 1 are quite close. Since the two models are all fit the data, it’s reasonable to choose the simpler one.

2) The structure of the subdimension model is similar to the Rasch testlet model, except it also takes correlation between subdimensions into account. As I know, most information from the responses is mainly used to estimate main ability and the rest information is used to estimate subdimensional latent traits. Thus, estimates of subdimensional latent traits usually are not accurate enough. Therefore, I guess that estimates of correlation for subdimension with one another are basically not precise, too.