Examining the test-taking process: A cognitive perspective on the discourse cloze test

Testing consists of making a generalization about a population on the basis of what can be observed in a sample (APA 1985). This principle underlies not only the composition of the sample of individuals to be tested, but also the selection of content for the testing instrument itself. Careful attention to the content of assessment tasks is needed to ensure that they are representative of the domain of tasks bout which generalizations are to be drawn (Bachman 1990, Messick 1989). However content representativeness extends beyond mere attention to the problems posed and materials provided in the test. What is more crucially important is the manner in which test tasks are performed. Attention to test-taking process is especially creucial and problematic in the testing of cognitive processing skills. The sample will only be representative if the cognitive processes involved in task performance are representative of those involved in performance of the domain of tasks in the real world (Alderson 1990). Thus the validation of tests involving cognitive processes will not be complete unless it includes some examination of the processes by which solutions to test tasks are reached. This line of thinking was propounded forty years ago by Lennon (1956) but has only recently been taken up with the application of ethnographic methods to the study of the test taking process. Cloze tests have strong theoretical foundations in theories of language processing and text sampling but the psycholinguistic construct underlying their use in language testing is based on armchair introspection rather than empirical evidence. Proponents of cloze as a measure of language proficiency (Oller and Conrad 1971, Oller 1973, Fotos 1991) and intersentential processing (Chihara et al 1977, Chavez-Bachman 1982, 1985, Porter 1983) have largely neglected to provide evidence from the perspective of the test taker. This paper looks at one typed of cloze from this perspective providing evidence of the validity of the measure based on the cognitive processes test takers employ in completing gaps. The paper presents the findings of a study of the processes employed by subjects engaged in a discourse cloze test (Deyes 1984, Levenston et al 1984). Think aloud protocols obtained from test –takers give indications of the reasoning employed in selecting items to complete gaps in the cloze passage and of the strategies employed in doing so. From these indications, cognitive processes are inferred. Comparing observed behaviours and inferred processes with those predicted in a model of ideal performance gives a measure of the construct validity both of the items and of the discourse cloze procedure in general. This introspective validation procedure reveals the test-taking behaviour of subjects engaged in doze tasks and provides a cognitive perspective on the vexed question of what cloze actually measures.