How impressionistic can holistic scoring be? A test-taker perspective

Most existing studies on L2 essay tests take a rater perspective focusing on rater characteristics and textual features, there are scarce research taking a test-taker perspective. This study investigated testtaker perceptions of an essay test and their impact on essay scores generated holistically, thereby drawing implications for score validity and washback. Around 880 Chinese test takers took an EFL writing test including an essay test and a translation task; they filled in a perception questionnaire immediately afterwards. Four raters marked their essays holistically. Factor analyses were conducted to reveal the underlying patterns of test-taker perceptions. Structural equation modelling was then applied to estimate the effects of perceptions on essay scores based on single and double ratings, and composite writing scores based on the essay and the translation task. The analysis found three sets of skills were perceived as important for essay tests, including the desirable writing skills to produce a good essay, risk-taking skills to impress raters, and defensive skills to avoid penalties. While the perceived writing skills did not make significant contribution to any of the three scores, the other two test-taking skills did, though their effects varied across raters. While raters tend to give higher scores to defensive test-takers, they may give lower scores to risk-taking ones. Depending on the way raters were paired, the effect pattern on single raters may or may not extend to scores based on double ratings. On the other hand, it extended to composite writing consistently, though the magnitude of the effects was attenuated. The implications of this study for score validity and washback are discussed at the end.