Statistical Consequences of Attributes Misspecification in the Rule Space Method
by Seongah Im & James E. Corter (20 11 ) .
Generally, cognitive diagnosis should proceed using the true Q-matrix representing complete and accurate specification of attribute-item involvement. However, the parameter estimations will be influenced by the Q-matrix misspecification. This paper investigates the statistical consequences of attributes misspecification in the rule space method for cognitively diagnostic measurement. Two types of attributes misspecifications examined are exclusion and inclusion. Attribute exclusion means that excluding an essential attribute while leading to elimination of some knowledge states. Thus, the examinees positioned near the elimination knowledge states might be either not classified or underclassified. Attribute inclusion means that including a superfluous attribute while leading to creation of knowledge states. Thus, the examinees positioned around the superfluously added knowledge states would be overclassified. The effects of order relations among attributes also investigated in this paper.
The results for investigation of the exclusion/inclusion attributes were that, first, when an essential attribute was excluded, the classification consistencies of examinees’ attribute mastery were lower than that when a superfluous attribute was included. Second, when an essential attribute was excluded, the estimated attribute mastery probabilities (AMPs) of examinees are underestimated; when a superfluous attribute is included, the estimated AMPs of examinees are overestimated. Third, when an essential attribute is excluded, the RMSEs of the estimated AMPs of examinees were larger than that when a superfluous attribute is included. In sum, the effects of excluding an essential attribute are more harmful to the classification of examinee’s attribute mastery and estimation of examinees’ AMPs, compared with the effects of including a superfluous attribute.
When an attribute is in an order relation with a misspecified attribute, the results are, first, when an excluded essential attribute is the subset of some remaining attributes, all the examinees’ AMPs on those attributes are underestimated or remain the same, and vice versa. Second, the effects of attribute inclusion are complementary to the effects of attribute exclusion. Third, RMSEs of the superset attribute because of misspecification of subset attributes were bigger than that of the subset attributes because of misspecification of the superset attributes.
Comments and Questions:
1. Minor revise is needed in the second paragraph in page 716.
Original sentence: An is referred to as a subset of Am and Am a superset of An.
Revise sentence: An is referred to as a superset of Am and Am a subset of An.
2. To verify the correctness of the Q-matrix is very important in cognitive diagnosis because of the examinees’ profiles are obtained from it. Based on this paper, we can understand that how the parameter estimations will be influenced by the Q-matrix misspecification.
3. Exclusion/inclusion attributes design is easy and intuitive to understand the parameter estimations are underestimated/overestimated. For example, when excluding an essential attribute likes removing some important information and leading to the parameter estimations are underestimated.
4. I don’t understand the results for order relation between attributes. For example, the expectation is that excluding/including an attribute will underestimated/overestimated the parameter estimations. The information of the superset should larger than that in subset. But when an essential subset attribute was excluded, the RMSEs of superset attributes were higher than that of subset attributes when a superset attribute was excluded.