Connie's readings and review

Least Square Distance Method of Cognitive Validation and Analysis for Binary Items Using Their Item Response Theory Parameters

Least Square Distance Method of Cognitive Validation and Analysis for Binary Items Using Their Item Response Theory Parameters

by HSU Chia Ling -
Number of replies: 0

    The cognitive structure of a test is typically defined as a set of cognitive attributes required to produce correct answers on the test items. It can provide cognitive diagnostic information to test developers, psychologists, educators and students for teaching strategies, constructing test items, and learning cognitive processes, respectively. All previous cognitively studies require item score information and do not focus on cognitive validation and analysis across individual items and ability levels. This study proposes an approach to validation and analysis of cognitive attributes required for correct answers on binary items or tasks by examining validity across fixed ability levels and individual items.

    The proposed method uses item response theory (IRT) estimates of the item parameters and the Q-matrix to estimate the probability on correct performance on each attribute across fixed ability levels on the logit scale. Based on this rationale, the attribute probability curves and diagnostic validation of the attributes for individual items can be obtained directly. The key component of this method is use the least squares distance method (LSDM) for estimating attribute probabilities. A unique feature of the LSDM is that is does not require performance data on individual items or cognitive attributes, as long as estimates of the item parameters are available through Rasch, 2PL, or 3PL calibration in IRT. The simulation results shown that, first, the smaller the LSD at an ability level, the better the cognitive attributes hold at that ability level. Second, the attribute probability curves of valid attributes should exhibit logical and substantively meaningful behavior in monotonicity, relative difficulty, and discrimination across ability levels. Third, the better the item characteristic curve recovery with the LSDM for an item, the better the required attributes hold for this item.

Comments:

      Validation of cognitive attributes is an important issue in cognitive diagnosis. This study introduced a common method (least squares distance) for validation of the cognitive attributes for individual items. The future study can be implied it to the polytomous items for practicing and understanding this method.