A Hierarchical Rater Model for Constructed Responses with a Signal Detection Rater Model
In the past, the conventional rater models connect person ability and rater severity into the same model. Under such model framework, precision of person ability would be increasing with increased by raters but not items. Actually, it is quite weird that for aforementioned problem exists. Patz et al. (2002) proposed two levels rater model to consider the conditions of different raters may score different items. However, the model could not provide other rater effects of central tendency and restriction of the range except rater precision and rater severity (or leniency). This paper is quite interesting that proposed a extend signal detection rater model to replace the model of level one of Patz et al. (2002). A real data was analyzed by proposed model and model of Patz et al. (2002).
Questions:
(1) For extend signal detection theory model, η are latent category parameters for raters. Here authors treated η as true values in the second level and which is indicator providing information for estimating person ability. I was wondering is it appropriate we ignore stand error of eta estimates. Another problem is that how could we compute stand error for η estimates since it actually is an ordinal data.
(2) Past studies illustrated that parameters for the level two were not recovery well due to two items (essays in present study). In practical application, items for essays writing would not likely to over than two. How can we to obtain well-recovery parameters for items in such real case?