Grol-Prokopczyk, Hanna
Working paper no. 2010-18
Abstract
In the past decade, anchoring vignettes have become an increasingly popular tool for identifying and correcting for group differences in use of subjective ordered response categories. However, existing techniques to maximize response consistency (the use of the same standards for self-ratings as for vignette-ratings), which center on matching vignette characters’ demographic characteristics to respondents’ own characteristics, appear at times to be ineffective or to pose interpretive difficulties. For example, respondents often appear to neglect instructions to treat vignette characters as age peers. Furthermore, when vignette characters are depicted as have the same sex as the respondent, interpretation of observed sex differences in rating style is rendered problematic. This paper applies two experimental manipulations to a national sample (n=1,765) to clarify best practices for enhancing response consistency. First, a comparison of ratings of same- and opposite-sex vignette characters suggests that, with occasional and avoidable exceptions, the sex of the respondent rather than the sex of the vignette character drives observed sex differences in rating style. Second, an analysis of two methods of highlighting vignette characters’ age shows that both yield better response consistency than previous, less prominent means. Implications for interpretation and design of anchoring vignette studies are discussed. In addition to methodological contributions, this paper represents the first fielding of general health vignettes to a national sample. Findings show significant differences in health-rating style across racial/ethnic groups, educational categories, and sex. Significant racial/ethnic differences in styles of rating political efficacy are also observed. These findings underscore the incomparability of unadjusted subjective self-ratings across demographic groups.