A Hierarchical Rater Model for Constructed Responses,
with a Signal Detection Rater Model
Lawrence T. DeCarlo
YoungKoung Kim
Matthew S. Johnson
The HRM-SDT model is introduced here which combine the Hierarchical Rater Model with the Signal Detection Rater Model. The main idea is that raters through providing information about the quality of an essay (or other CR), in turn, provides information about the examinee’s proficiency. There are two levels:
The first level: the raters’ scores are ordinal indicators of the “true” category that an essay belongs to—a signal detection model
The second level: the latent categories are ordinal indicators of examinees’ proficiency—IRT model-GPCM
The HRM-SDT model is applied to data from a large-scale assessment and is shown to provide a useful summary of various aspects of the raters’ performance.
Comment：
In the formulation (2), it seems that parameter c is used for judging categories. Before that, d which is the distance between two distributions has to be computed. If d is larger, the decision is easier to make. Which I think during the computing it is not a constant it is been translated.
It seems no general indices for comparing which model is better?
The model can be used in the writing section of TOEFL test. Through formulation, the IRT scores of GPCM can be transform to sum scores. While one problem is that sometimes the composition can’t be finished in time due to the time limit. Can this model use to deal with this kind of situation considering speed？