This article introduces a hierarchical model that deals with rater effect, specifically a signal detection rater effect, in items that elicit constructed response, in the authors’ term: HRM-SDT model. The authors try to conceptualize the scoring of CR items by raters as a signal detection process, where a rater try to detect which category an CR belongs to, suppose there is a latent continuum of categories against which a rater gives a particular score according to his judgment of the quality of the response. This article incorporates the concept latent class into the SDT model to capture the “catalog of rater effects”. The proposed model has two layers, one of which rater responses load directly on examinee proficiency in an IRT approach. This approach can allow separation of rater effects and item effects. Practically, besides differentiate more capable examinees and less capable examinees, this model helps differentiate more proficient raters from less proficient rater.
Comments:
Is it possible/ useful to make an extension that incorporates latent classes within a rater to indicate the non-invariant severity of any particular rater?