Silk purses from sows' ears? Making measures from teacher judgements

The prospect of increased mandated achievement testing in Australian schools has the potential to relegate into insignificance the professional judgements of classroom teachers. This paper reports the first steps of a two-stage project to foreground teachers' judgements in the assessment process. Firstly, we investigate the extent to which teachers' assessments of their students satisfy the strict measurement requirements of the Rasch model. Secondly, we attempt to integrate the ability estimates derived from teacher judgements into the more typical quantitative results derived from standardized testing. Two data sets (from 1997 and 1998) record teacher assessments of the development of approximately 10,000 primary school students for each calendar year using the Australian National Profiles (reported by Rothman, AARE, 1998). As a separate unrelated event all students in Years 3 and 5 were also assessed using a Literacy scale using the NSW Basic Skills test. With the recent approval and support of the SA school system, the 'intersect' of the two sets has been matched at Years 3 and 5, in English in 1997 and Mathematics in 1998. (1000 teacher assessments at each Year level matched with 12,000 test assessments). Files of approximately 700 students with both assessments have been created for the 4 data sets to explore (a) the development of a measurement scale of teacher assessments and (b) how well the two approaches to assessment of students match. Measurement error based on the Rasch model has been estimated for both the test and for teacher judgements. Teachers vary considerably in their observational skills, their understanding of learning, their comfort with the ambiguous profile scales, their personal specific knowledge of the randomly selected students and their confidence that they can use criteria described scales. Many teachers' assessments correlate well with the test, some differ widely. The paper then speculates on how to improve the skill of teachers in using the latent scales established in test analysis as a support to the measurement of growth in the classroom and, more generally, how to use criteria scales in formative assessment.