A mixture Rasch facets model for raters' illusory halo effects

Kuan-Yu JIN; Ming Ming CHIU

A mixture Rasch facets model for raters' illusory halo effects

Kuan-Yu JIN
, Ming Ming CHIU

Department of Special Education and Counselling (SEC)

Research output: Contribution to conference › Papers

Abstract

A rater's overall impression of a ratee's essay (or other assessment) can influence ratings on multiple criteria to yield excessively similar ratings (halo effect). Hence, we introduce and test a mixture Rasch facets model for halo effects (MRFM-H) that distinguishes true versus illusory halo effects and classifies normal and halo raters. In a simulation study, when raters assessed enough ratees, MRFM-H accurately identified halo raters. Also, more rating criteria increased classification accuracy. Ignoring halo effects (via a simpler model) biased parameters for evaluation criteria and rater severity but not ratee assessments. MRFM-H application to three empirical datasets showed (a) experienced raters' illusory halo effects, (b) fewer illusory halo effects with more criteria; and (c) more versus less informative survey responses. Copyright © 2021 AERA21.

Original language	English
Publication status	Published - Apr 2021
Event	2021 Virtual Annual Meeting of American Educational Research Association: "Accepting Educational Responsibility" - , United States Duration: 08 Apr 2021 → 12 Apr 2021 https://www.aera.net/Events-Meetings/2021-Annual-Meeting

Conference

Conference	2021 Virtual Annual Meeting of American Educational Research Association: "Accepting Educational Responsibility"
Abbreviated title	AERA 2021
Country/Territory	United States
Period	08/04/21 → 12/04/21
Internet address	https://www.aera.net/Events-Meetings/2021-Annual-Meeting

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 4 Quality Education

@conference{30ac9d543ed14cd89b6aa7516bfde25a,

title = "A mixture Rasch facets model for raters' illusory halo effects",

abstract = "A rater's overall impression of a ratee's essay (or other assessment) can influence ratings on multiple criteria to yield excessively similar ratings (halo effect). Hence, we introduce and test a mixture Rasch facets model for halo effects (MRFM-H) that distinguishes true versus illusory halo effects and classifies normal and halo raters. In a simulation study, when raters assessed enough ratees, MRFM-H accurately identified halo raters. Also, more rating criteria increased classification accuracy. Ignoring halo effects (via a simpler model) biased parameters for evaluation criteria and rater severity but not ratee assessments. MRFM-H application to three empirical datasets showed (a) experienced raters' illusory halo effects, (b) fewer illusory halo effects with more criteria; and (c) more versus less informative survey responses. Copyright {\textcopyright} 2021 AERA21.",

author = "Kuan-Yu JIN and CHIU, \{Ming Ming\}",

year = "2021",

month = apr,

language = "English",

note = "2021 Virtual Annual Meeting of American Educational Research Association: {"}Accepting Educational Responsibility{"}, AERA 2021 ; Conference date: 08-04-2021 Through 12-04-2021",

url = "https://www.aera.net/Events-Meetings/2021-Annual-Meeting",

}

TY - CONF

T1 - A mixture Rasch facets model for raters' illusory halo effects

AU - JIN, Kuan-Yu

AU - CHIU, Ming Ming

PY - 2021/4

Y1 - 2021/4

N2 - A rater's overall impression of a ratee's essay (or other assessment) can influence ratings on multiple criteria to yield excessively similar ratings (halo effect). Hence, we introduce and test a mixture Rasch facets model for halo effects (MRFM-H) that distinguishes true versus illusory halo effects and classifies normal and halo raters. In a simulation study, when raters assessed enough ratees, MRFM-H accurately identified halo raters. Also, more rating criteria increased classification accuracy. Ignoring halo effects (via a simpler model) biased parameters for evaluation criteria and rater severity but not ratee assessments. MRFM-H application to three empirical datasets showed (a) experienced raters' illusory halo effects, (b) fewer illusory halo effects with more criteria; and (c) more versus less informative survey responses. Copyright © 2021 AERA21.

AB - A rater's overall impression of a ratee's essay (or other assessment) can influence ratings on multiple criteria to yield excessively similar ratings (halo effect). Hence, we introduce and test a mixture Rasch facets model for halo effects (MRFM-H) that distinguishes true versus illusory halo effects and classifies normal and halo raters. In a simulation study, when raters assessed enough ratees, MRFM-H accurately identified halo raters. Also, more rating criteria increased classification accuracy. Ignoring halo effects (via a simpler model) biased parameters for evaluation criteria and rater severity but not ratee assessments. MRFM-H application to three empirical datasets showed (a) experienced raters' illusory halo effects, (b) fewer illusory halo effects with more criteria; and (c) more versus less informative survey responses. Copyright © 2021 AERA21.

M3 - Papers

T2 - 2021 Virtual Annual Meeting of American Educational Research Association: "Accepting Educational Responsibility"

Y2 - 8 April 2021 through 12 April 2021

ER -

A mixture Rasch facets model for raters' illusory halo effects

Abstract

Conference

UN SDGs

Fingerprint