Comparison of rater’s reliability on perceptual evaluation of different types of voice sample

Objective/Hypothesis: The objective of this study was to determine whether different types of voice samples affect rater reliability and which type of sample can be rated most reliably, with particular reference to two types of connected speech - passage reading and conversational speech. 

Study Design: Prospective reliability study. 

Methods: One hundred fifty voice samples from 40 speakers were presented to 14 speech pathologists experienced in managing voice disorders. Each speaker contributed three types of voice samples: sustained vowels, passage reading, and conversational speech. Ratings were made on four vocal parameters - overall severity, roughness, breathiness, and strain - on a 10-point equal-appearing interval scale. 

Results: Differences in intrarater reliability across the three types of voice samples were noted. Higher intrarater reliability was achieved with connected speech than with sustained vowel samples. Interrater reliability showed no statistically significant difference across the three types but increased with the severity of dysphonia. 

Conclusions: This study reveals that raters give internally more reliable ratings for connected speech samples. Results also indicate that voices with severe disorders appear to be rated more reliably. Copyright © 2012 The Voice Foundation.