Design, Evaluation, and Analysis of Questionnaires for Survey Research, 2nd Edition

9

CRITERIA FOR THE QUALITY OF SURVEY MEASURES

In Parts I and II, we have seen that the development of a survey item demands making choices concerning the structure of the item and the data collection procedure. Some of these choices follow directly from the aim of the study, such as the choice of the topic of the survey item(s) (church attendance, neighborhood, etc.) and the concept measured by the request for an answer (evaluations, norms, etc.). But there are also many choices that are not fixed and these choices will influence the quality of the survey items. They have to do with the formulation of the request, the response scales, and any additional components such as introduction, motivation, position in the questionnaire, and the mode of data collection. Therefore, it is highly desirable to have some information about the quality of a survey item before it is used in the field.

Several procedures have been developed to evaluate survey items before they are used in the final survey. The oldest and most commonly used approach is, of course, the use of pretests and debriefing of the interviewers regarding any problems that may arise in the questionnaire. Another approach, suggested by Belson (1981), is to ask people during a pretest, after they have answered a request for an answer, how they interpreted the different concepts in the survey item while they were answering the requests. A third approach is the use of “think aloud” protocols during interviews. A fourth approach is ...