Main purpose of this paper is to evidence that the Tailored method performs well in estimating item parameters using the RASCH model. In order to prove Tailored method is good at controlling guessing when estimating item difficulties, this study employed 3 analysis, which were referred to as: First, Tailored, and Anchored analysis.
Procedure:
In the first step (First analysis): All original data were analyzed using RASCH model, without any specific constraint. Results showed that, difficulties for easy items were over-estimated, when difficulties for hard items were under-estimated. The poor results were probably due to the effects of guessing.
In the second step (Tailored analysis): If probability of correct response to an item is less than .30 for a person, then that response would be converted to missing data for the tailored analysis. After converting, remained data were analyzed again. Results for tailored method represented good recovery of item difficulties.
In the third step (Anchored analysis): six most easy items used as anchored items with fixed mean, all original data were re-analyzed. Andersen’s theorem was employed in this step to test whether estimations of item difficulties would be influenced by guessing. It was found that all hard items, whose real difficulties larger than .88, had been under-estimated and significantly affected by guessing. In other words, the tailored method was confirmed again to be appropriate method for estimating item difficulties with RASCH model.
Advantages:
1. Using tailored method, RASCH method could come up with more precise item estimation.
2. Don’t need to employ more complicated models, which take guessing into account.
Questions:
1. Although the tailored method been proved to be more accurate, the cut-point for tailored analysis was arbitrary. Most important, the probability for a correct response was calculated based on the results of the First analysis, however, the First analysis’s results showed poor recovery. Is it suitable to convert some data into missing based on these results?
2. When choosing the anchor items, why not use items with moderate difficulties? Those are around 0? As their SE are smaller than the so-called easy items.
3. For Figure 4, the square represent results of anchored, it seems they are over-estimated. Is there any error in this figure’s X and Y scales’ names? As anchored method should show under-estimated.
but i think selection of cut-point is similar to the question that, why always set the alpha level as .05?