ARC Laboratory Sharing

Detecting Response Styles and Faking in Personality and Organizational Assessments by Mixed Rasch Models

This chapter made an overview of applying mixed Rasch models to personality and organizational assessments. Contrary to ability assessment, respondents may know the “correct” answer or to make a specific response due to the fact that they want to follow socially desirable answer even if they don’t really think about themselves as persons responding to the type of person they belong to. In such situation, clinical psychologists and employers cannot make a good diagnosis and arrangement for patients and employees. The authors here introduced mixed Rasch models to deal with such problems aforementioned and also shown the information and advantages the mixed Rasch models can provide. Several empirical datum were fit by IRT models to assess which model has the best fit according to ACI and CACI indices. Results revealed that response styles, faking, and structure difference in item parameters can be interpreted meaningful by the mixed Rasch models more than the conventional models.

Main idea of the new method

Empirical datum on dichotomous items and polytomous items were examined through the mixed Rasch models and HYBRID models. For the dichotomous items, respondents were separated into two subgroups of response patterns which are yea-sayers and nay-sayers. However, in this empirical data, it’s not easy to find response style classes because of all participants were allowed to decide to take the questionnaire by themselves. For polytomous items, the authors shown some studies in the past and that those studies were to examine unmotivated personality data, measurement invariance, faking, and response styles in anger expression. Results illustrated that the mixed models could make more informative conclusion, for example, item threshold would be disorder for those who were faking.

Comments, Questions and Future Study

1) As Prof Wang’ comments last week, if the analysis on the mixed Rasch model only shows that there are two classes which are high-ability class and low-ability class. This kind of analysis is useless due to the fact that we also can get such information from individual’s ability. Similarly, this conclusion easily appears on test of dichotomous items. Attribute yea-sayers and nay-sayers to two different classes have the same confounding of latent trait. It would be much meaningful if the subgroups are independent from latent trait, such as faking and no-faking.

2) This model can be used to detect faking response pattern in the future studies.

3) The authors mentioned that this kind of studies is similar DIF concept. Thus, we can apply the mixed Rasch models to detect DIF. On my opinion, only DIF items are a lot, so that we can accurately distinguish unknow membership. It’s kind of tricky, because we don’t want there are too many DIF items in real data. Therefore, I’m quite confused that how to apply the mixed models on DIF detecting.

The Effect of Person Misfit on Classification Decisions
This paper investigated that applied person fit statistics to improve the correct rate of classification decisions. A series of simulations were conducted to illustrate the effect of person misfit on classification decisions. Several independent variables were manipulated for the simulations: cut-point for decision judgment, test length, person fit statistic, type of aberrant behavior, estimation method, and sample size. The dependent variable was correct decision rate of normal group and misfit group. The results shown that the effect of misfit responses would affect item and person estimation and cut-point obviously affect the correct decision rate, particularly for cut-point was set to be -1 and 1.

Comments, Questions and Future Study
1) Aberrant behaviors such like guessing and cheating, which are mainly to be investigated. However, researchers generated such responses violations with different manipulation in person fit studies. Thus, it is hard to compare results on different studies on the same basis. Consequently, suggestions and results are only dependent on their studies. I think we need to do some literature of cognitive psychology on guessing and cheating behaviors, then try to design a more reasonable simulation scenarios. That would help different studies can be comparative if they adopt the same design frame.
2) The authors basically used person fit to improve the correct decision rate on classification testing. I think we can extend the study to computerized classification testing. However, these two testing have different algorithms, it would be a problem how to chose which person fit statistic to be used.
3) The misfit responses would affect item and person estimation, then person fit statistics might be affected by biased parameter estimations. We can try to apply purification procedure to improve estimation into the study. That would mainly reduce the effect of misfit on estimation.

Testing Person Fit in Cognitive Diagnosis

This paper developed likelihood-based ratio testing to detect two types of aberrant responses for cognitive diagnosis model (CDM). Two CDMs were introduced: the first one is DINA model which is the simplest one to depict responses that result from person attributes; and the second one is RUM which is model responses with more than two probabilities comparing with DINA model. Three simulations were conducted by the authors and the first one was to evaluate marginal and joint likelihood ratio testing under true item parameters, estimated item parameters, and item parameters were perturbed within 95% confidence intervals. Moreover, the last two simulations were to investigate two types of aberrant response of spuriously high scores that result from multiple strategies are available and spuriously low scores that is caused by Q-matrix is partly missing. Finally, a real data was analyzed by DINA and RUM models. Results illustrated that proposed methods performed adequately well for detecting persons with aberrant responses.

Main ideas of the present paper

• DINA model (eq.1 and 2)
• RUM (eq.1 and 3)
• Aberrant response or not (eq.4) 
• Joint likelihood ratio testing (eq.5 and 7)
• Marginal likelihood ratio testing (eq 6 and 8)

Comments, Questions and Future Study

1) Conventionally, person fit statistics have been adopted for continuous IRT models. The authors put main ideas from past researches and developed adequate method for CDM models and found the methods performed quite well.

2) From the first simulation, it illustrated that it had the best performance when true item parameters were known. It provided a hint for us to think about that more accurate estimated attributes may help to improve the detection of aberrant responses. As the iterate strategy we adopted under conventional IRT model, the strategy may also benefit for using under CDM.

A Didactic Presentation of Snijders’ lz* Index of Person Fit with Emphasis on Response Model Selection and Ability Estimation

Conventional lz index has been popular and adopted in the studies of person fit. However, the index only follow the standardize normal distribution when computing by true ability. In practice, we can obtain estimated ability from collecting data, thus lz doesn’t fulfill the assumption of standardize normal distribution. As a result, pre-specific nominal level may not be achieved in such situation. Snijders’ lz had been developed to resolve the problem, however, it has not been common used nowadays. The main reason is that the procedure is not easy to follow, so that the authors tried to introduce a didactic technique on Snijders’ lz method with more concrete description. The Snijders’ lz follows standardize normal distribution when estimated ability is used on index computation if ability method meets equation 14 of the paper. As my point of view, it is really better than conventional one under the scenario of item parameters known. In the reality, item parameters and person parameter are estimated simultaneously. Thus, parameters may affected by noise of aberrant behaviors, so that Snijders’ lz would not follows the property of standardize normal distribution.