19 Survey Research Literature
This page provides references in readings for survey research. This page is continually being revised and updated to reflect new resources. Do you have suggestions for readings we should add to this page? Send us an email with your suggestions and a link to this page!
Quicklinks:
- Introduction to Survey Research
- Data Collection Literature
- Sample Design and Selection Literature
- Questionnaire Development and Layout Literature
- Data Collection and Processing Literature
- Data Analysis Literature
20 Introduction to Survey Research Literature
Krosnick, Jon A. 1999. “Survey Research.” Annual Review of Psychology. 50:537-567.AUTHOR ABSTRACT
For the first time in decades, conventional wisdom about survey methodology is being challenged on many fronts. The insights gained can not only help psychologists do their research better but also provide useful insights into the basics of social interaction and cognition. This chapter reviews some of the many recent advances in the literature, including the following: New findings challenge a long-standing prejudice against studies with low response rates; innovative techniques for pretesting questionnaires offer opportunities for improving measurement validity; surprising effects of the verbal labels put on rating scale points have been identified, suggesting optimal approaches to scale labeling; respondents interpret questions on the basis of the norms of everyday conversation, so violations of those conventions introduce error; some measurement error thought to have been attributable to social desirability response bias now appears to be due to other factors instead, thus encouraging different approaches to fixing such problems; and a new theory of satisficing in questionnaire responding offers parsimonious explanations for a range of response patterns long recognized by psychologists and survey researchers but previously not well understood.BOOK DESCRIPTION
This new edition of Survey Methodology continues to provide a state-of-the-science presentation of essential survey methodology topics and techniques. The volume’s six world-renowned authors have updated this Second Edition to present newly emerging approaches to survey research and provide more comprehensive coverage of the major considerations in designing and conducting a sample survey.
Key topics in survey methodology are clearly explained in the book’s chapters, with coverage including sampling frame evaluation, sample design, development of questionnaires, evaluation of questions, alternative modes of data collection, interviewing, nonresponse, post-collection processing of survey data, and practices for maintaining scientific integrity. Acknowledging the growing advances in research and technology, the Second Edition features:
Updated explanations of sampling frame issues for mobile telephone and web surveys New scientific insight on the relationship between nonresponse rates and nonresponse errors
Restructured discussion of ethical issues in survey research, emphasizing the growing research results on privacy, informed consent, and confidentiality issues
The latest research findings on effective questionnaire development techniques
The addition of 50% more exercises at the end of each chapter, illustrating basic principles of survey design
An expanded FAQ chapter that addresses the concerns that accompany newly established methods
BOOK DESCRIPTION
Surveys enjoy great ubiquity among data collection methods in social research: they are flexible in questioning techniques, in the amount of questions asked, in the topics covered, and in the various ways of interactions with respondents. Surveys are also the preferred method by many researchers in the social sciences due to their ability to provide quick profiles and results. Because they are so commonly used and fairly easy to administer, surveys are often thought to be easily thrown together. But designing an effective survey that yields reliable and valid results takes more than merely asking questions and waiting for the answers to arrive. Geared to the non-statistician, the Handbook of Survey Methodology in Social Sciences addresses issues throughout all phases of survey design and implementation. Chapters examine the major survey methods of data collection, providing expert guidelines for asking targeted questions, improving accuracy and quality of responses, while reducing sampling and non-sampling bias. Relying on the Total Survey Error theory, various issues of both sampling and non-sampling sources of error are explored and discussed. By covering all aspects of the topic, the Handbook is suited to readers taking their first steps in survey methodology, as well as to those already involved in survey design and execution, and to those currently in training. Featured in the Handbook: • The Total Survey Error: sampling and non-sampling errors. • Survey sampling techniques. • The art of question phrasing. • Techniques for increasing response rates • A question of ethics: what is allowed in survey research? • Survey design: face-to-face, phone, mail, e-mail, online, computer-assisted.? • Dealing with sensitive issues in surveys. • Demographics of respondents: implications for future survey research. • Dealing with nonresponse, and nonresponse bias The Handbook of Survey Methodology in Social Sciences offers how-to clarity forresearchers in the social and behavioral sciences and related disciplines, including sociology, criminology, criminal justice, social psychology, education, public health, political science, management, and many other disciplines relying on survey methodology as one of their main data collection tools.21 Data Collection Literature
Bauman, Adrian, P. Phongsavan, A. Cowle, E. Banks, L. Jorm, K. Rogers, B. Jalaludin, and A. Grunseit. 2016. “Maximising Follow-up Participation Rates in a Large Scale 45 and Up Study in Australia.” Emerging Themes in Epidemiology 13:6.AUTHOR ABSTRACT
The issue of poor response rates to population surveys has existed for some decades, but few studies have explored methods to improve the response rate in follow-up population cohort studies. A sample of 100,000 adults from the 45 and Up Study, a large population cohort in Australia, were followed up 3.5 years after the baseline cohort was assembled. A pilot mail-out of 5000 surveys produced a response rate of only 41.7 %. This study tested methods of enhancing response rate, with three groups of 1000 each allocated to (1) receiving an advance notice postcard followed by a questionnaire, (2) receiving a questionnaire and then followup reminder letter, and (3) both these strategies.The enhanced strategies all produced an improved response rate compared to the pilot, with a resulting mean response rate of 53.7 %. Highest response was found when both the postcard and questionnaire reminder were used (56.4 %) but this was only significantly higher when compared to postcard alone (50.5 %) but not reminder alone (54.1 %). The combined approach was used for recruitment among the remaining 92,000 participants, with a resultant further increased response rate of 61.6 %. Survey prompting with a postcard and a reminder follow-up questionnaire, applied separately or combined can enhance follow-up rates in large scale survey-based epidemiological studies.AUTHOR ABSTRACT
Increasingly, American households are choosing to forgo ownership of landline telephones in favor of cell phones. Presently, more than 25% of Hispanics now only own a cell phone. Concern about potential bias from non-coverage of this “cell-only” population in traditional general population RDD (random digit dial) telephone interviewing has been a particular focus among the survey research community. Because Hispanics lead all ethnic and racial groups in the percentage who are now cell-only, it is critical to understand the impact this has on research of Hispanics, which to date is still largely accomplished using traditional landline telephone survey methods. This article explores the extent to which non-coverage of cell-only Hispanics leads to raw bias in landline telephone surveys of Hispanics and the degree to which survey weights may correct for such bias.AUTHOR ABSTRACT
This article examines the problem of response error in survey earnings data. Comparing workers’ earnings reports in the U.S. Census Bureau’s Survey of Income and Program Participation (SIPP) to their detailed W-2 earnings records from the Social Security Administration, we employ ordinary least squares (OLS) and quantile regression models to assess the effects of earnings determinants and demographic variables on measurement errors in 2004 SIPP earnings in terms of bias and variance. Results show that measurement errors in earnings are not classical, but mean-reverting. The directions of bias for subpopulations are not constant, but varying across levels of earnings. Highly educated workers more correctly report their earnings than less educated workers at higher earnings levels, but they tend to overreport at lower earnings levels. Black workers with high earnings underreport to a greater degree than comparable whites, while black workers with low earnings overreport to a greater degree. Some subpopulations exhibit higher variances of measurement errors than others. Blacks, Hispanics, high school dropouts, part-year employed workers, and occupation ‘‘switchers’’ tend to misreport—both over- and underreport—their earnings rather than unilaterally in one direction. The implications of our findings are discussed.AUTHOR ABSTRACT
With increasing migration and linguistic diversification in many countries, survey researchers and methodologists should consider whether data provided by individuals with variable levels of command of the survey language are of the same quality. This paper examines the question of whether answers from resident foreign respondents who do not master available survey languages may suffer from problems of comprehension of survey items, especially items that are more complicated in terms of content and/or form. In addition, it addresses the extent to which motivation may affect the response quality of resident foreigners. We analyzed data from two large-scale surveys conducted in Switzerland, a country with three national languages and a burgeoning foreign population, employing a set of dependent measures of response quality, including don’t know responses, extreme responding, mid-5 responding, recency effects, and straight-lining. Results show overall poorer response quality among foreigners, and indicate that both reduced language mastery and motivation among foreigners are relevant factors. This is especially true for foreign groups from countries that do not share a common language with those spoken in Switzerland. A general conclusion is that the more distant respondents are culturally and linguistically from the majority mainstream within a country, the more their data may be negatively affected. We found that more complex types of questions do generally lead to poorer response quality, but to a much lesser extent than respondent characteristics, such as nationality, command of the survey language, level of education, and age.AUTHOR ABSTRACT
The quality of data collected in survey research is usually indicated by the response rate; the representativeness of the sample, and; the rate of completed questions (item-response). In attempting to improve a generally declining response rate in surveys considerable efforts are being made through follow-up mailings and various types of incentives. This study examines effects of including a scratch lottery ticket in the invitation letter to a survey. Questionnaires concerning oral health were mailed to a random sample of 2,400 adults. A systematically selected half of the sample (1,200 adults) received a questionnaire including a scratch lottery ticket. One reminder without the incentive was sent. The incentive increased the response rate and improved representativeness by reaching more respondents with lower education. Furthermore, it reduced item nonresponse. The initial incentive had no effect on the propensity to respond after the reminder. When attempting to improve survey data, three issues become important: response rate, representativeness, and item-response. This study shows that including a scratch lottery ticket in the invitation letter performs well on all the three.22 Sample Design and Selection Literature
Alwin, Duane F. 2014. “Investigating Response Errors in Survey Data.” Sociological Methods & Research, 43(1), 3-14.AUTHOR ABSTRACT
This special issue of Sociological Methods & Research contributes to recent trends in studies that exploit the availability of multiple measures in sample surveys in order to detect the level and patterning to measurement errors. Articles in this volume focus on topics in one of (or some combination of) the three areas: (1) those that develop and test theoretical hypotheses regarding the behavior of measurement errors under specific conditions of measurement, (2) those that focus on the methodological problems encountered in the design of data collection permitting the estimation of measurement models, and (3) those that focus on the evaluation of existing models for detecting and quantifying the nature of measurement errors. The designs included in these investigations include those that incorporate follow-up probes, record-check studies, multitrait-multimethod designs, longitudinal designs, and latent class models for assessing measurement errors for categorical variables.AUTHOR ABSTRACT
Background: The issue of poor response rates to population surveys has existed for some decades, but few studies have explored methods to improve the response rate in follow-up population cohort studies.
Methods: A sample of 100,000 adults from the 45 and Up Study, a large population cohort in Australia, were followed up 3.5 years after the baseline cohort was assembled. A pilot mail-out of 5000 surveys produced a response rate of only 41.7 %. This study tested methods of enhancing response rate, with three groups of 1000 each allocated to (1) receiving an advance notice postcard followed by a questionnaire, (2) receiving a questionnaire and then followup reminder letter, and (3) both these strategies.
Results: The enhanced strategies all produced an improved response rate compared to the pilot, with a resulting mean response rate of 53.7 %. Highest response was found when both the postcard and questionnaire reminder were used (56.4 %) but this was only significantly higher when compared to postcard alone (50.5 %) but not reminder alone (54.1 %). The combined approach was used for recruitment among the remaining 92,000 participants, with a resultant further increased response rate of 61.6 %.
Conclusions: Survey prompting with a postcard and a reminder follow-up questionnaire, applied separately or combined can enhance follow-up rates in large scale survey-based epidemiological studies.
Keywords: Epidemiological studies, Follow-up, Response rates, Recruitment strategiesAUTHOR ABSTRACT
This study assessed the accuracy of telephone and Internet surveys of probability samples and Internet surveys of non-probability samples of American adults by comparing aggregate survey results against benchmarks. The probability sample surveys were consistently more accurate than the non-probability sample surveys, even after post-stratification with demographics. The non-probability sample survey measurements were much more variable in their accuracy, both across measures within a single survey and across surveys with a single measure. Post-stratification improved the overall accuracy of some of the non-probability sample surveys but decreased the overall accuracy of others. Higher completion and response rates of the surveys were associated with less accuracy. Accuracy did not appear to change from 2004/2005 to 2009 for any of the methods, and these conclusions are reinforced by data collected in 2008 as well. These results are consistent with the conclusion that non-probability samples yield data that are neither as accurate as nor more accurate than data obtained from probability samples.23 Questionnaire Development and Layout Literature
Bradley, Kelly D., Michael R. Peabody, Kathryn S. Akers, and Nichole M Knutson. 2015. “Rating Scales in Survey Research: Using the Rasch Model to Illustrate the Neutral Middle Category Measurement Flaw.” Survey Practice 8(2).AUTHOR ABSTRACT
The quality of the instrument used in the measurement process of survey data is fundamental to successful outcomes. Issues regarding content and structure are primary during instrument development, but the rating scale is just as important. Specifically for Likert-type questionnaires, the words used to describe rating categories and the placement of a neutral or not sure category is at the core of this measurement issue. This study utilizes the Rasch model to assess the quality of an instrument and structure of the rating scale for a typical data set collected at an institution of higher education. The importance of category placement and an evaluation of the use of a neutral middle category for Likert-type survey data is highlighted in this study.AUTHOR INTRODUCTION
The heart of a survey is its questionnaire. Drawing a sample, hiring, and training interviewers and supervisors, programming computers, and other preparatory work is all in service of the conversation that takes place between researchers and respondents. Survey results depend crucially on the questionnaire that scripts this conversation (irrespective of how the conversation is mediated, e.g., by an interviewer or a computer). To minimize response errors, questionnaires should be crafted in accordance with best practices.
Recommendations about best practices stem from experience and common lore, on the one hand, and methodological research, on the other. In this chapter, we first offer recommendations about optimal questionnaire design based on conventional wisdom (focusing mainly on the words used in questions), and then make further recommendations based on a review of the methodological research (focusing mainly on the structural features of questions).
We begin our examination of the methodological literature by considering open versus closed questions, a difference especially relevant to three types of measurement: (1) asking for choices among nominal categories (e.g., ‘‘What is the most important problem facing the country?’’), (2) ascertaining numeric quantities (e.g., ‘‘How many hours did you watch television last week?’’), and (3) testing factual knowledge (e.g., ‘‘Who is Joseph Biden?’’).
Next, we discuss the design of rating scales. We review the literature on the optimal number of scale points, consider whether some or all scale points should be labeled with words and/or numbers, and examine the problem of acquiescence response bias and methods for avoiding it. We then turn to the impact of response option order, outlining how it varies depending on whether categories are nominal or ordinal and whether they are presented visually or orally.
After that, we assess whether to offer ‘‘don’t know’’ or no-opinion among a question’s explicit response options. Next we discuss social desirability response bias (a form of motivated misreporting) and recall bias (a form of unmotivated misreporting), and recommend ways to minimize each. Following that, we consider the ordering of questions within a questionnaire and then discuss methods for testing and evaluating questions and questionnaires. Finally, we offer two more general recommendations to guide questionnaire development.AUTHOR ABSTRACT
Abstract Although Likert scales in agree-disagree (A/D) format are popular in surveys, the data quality yielded by them is controversial among researchers. Recognizing the measurement issues involved with the A/D format, researchers have developed other question formats to measure attitudes. In this study, we focused on an alternative question type, the item-specific (IS) question, which asks the respondent to choose an option that best describes his or her attitude. Using political efficacy items from the American National Election Studies (ANES), we compare extreme response style (ERS) between A/D and IS scales. Latent class factor analysis showed that ERS exists in both A/D and IS scale formats, but differs slightly across the two. Also, when analyzing ERS within subjects across two waves, there is only a single ERS for both question formats, after controlling for the correlation within respondents. The last finding suggests that ERS is a stable characteristic.AUTHOR ABSTRACT
This article provides a summary of the literature’s suggestions on survey design research. In doing so, it points researchers toward question formats that appear to yield the highest measurement reliability and validity. Using the American National Election Studies as a starting point, it shows the general principles of good questionnaire design, desirable choices to make when designing new questions, biases in some question formats and ways to avoid them, and strategies for reporting survey results. Finally, it offers a discussion of strategies for measuring voter turnout in particular, as a case study that poses special challenges. Scholars designing their own surveys should not presume that previously written questions are the best ones to use. Applying best practices in questionnaire design will yield more accurate data and more accurate substantive findings about the nature and origins of mass political behavior.AUTHOR ABSTRACT
The assumption that relationships between variables are unaffected by formal features of the measurement indicators has been shown to hold for most associations between attitudes and other variables, where the other variables are elf/ier background characteristics or other attitudes. This article tests the assumption of “form-resistant correlations” in the context of studying social change, where the other variable is time. It examines whether alternate forms of the same attitude item behave in similar fashion across time.AUTHOR ABSTRACT
Two quite different reasons for using open-ended as opposed to close-ended questions can be distinguished. One is to discover the responses that individuals give spontaneously; the other is to avoid the bias that may result from suggesting responses to individuals. However, open-ended questions also have disadvantages in comparison to close-ended, such as the need for extensive coding and larger item non-response. While this issue has already been well researched for traditional survey questionnaires, not much research has been devoted to it in recently used Web questionnaires. We therefore examine the differences between the open-ended and the close-ended question form in Web questionnaires by means of experiments within the large-scale RIS 2001 Web survey. The question “What is the most important, critical problem the Internet is facing today?” was asked in an open-ended and two close-ended question forms in a split-ballot experiment. The results show that there were differences between question forms in univariate distributions, though no significant differences were found in the ranking of values. Close-ended questions in general yield higher percentages than open-ended question for answers that are identical in both question forms. It seems that respondents restricted themselves with apparent ease to the alternatives offered on the close-ended forms, whereas on the open-ended question they produced a much more diverse set of answers. In addition, our results suggest that open-ended questions produce more missing data than close-ended. Moreover, there were more inadequate answers for open-ended question. This suggests that open-ended questions should be more explicit in their wording (at least for Web surveys, as a self administered mode of data collection) than close-ended questions, which are more specified with given response alternatives.AUTHOR ABSTRACT
Although agree–disagree (AD) rating scales suffer from acquiescence response bias, entail enhanced cognitive burden, and yield data of lower quality, these scales remain popular with researchers due to practical considerations (e.g., ease of item preparation, speed of administration, and reduced administration costs). This article shows that ifresearchers want to use AD scales, they should offer 5 answer categories rather than 7 or 11, because the latter yield data of lower quality. This is shown using data from four multitrait-multimethod experiments implemented in the third round of the European Social Survey. The quality of items with different rating scale lengths were computed and compared.AUTHOR ABSTRACT
Although Agree/Disagree (A/D) rating scales are hugely popular in the social sciences, a large body of research conducted during more than five decades has documented the bias that results from acquiescence in responses to these items. This may be a reason to prefer questions with Item Specific (IS) response options, but remarkably little research has explored whether responses to A/D rating scale questions are indeed of lower quality than responses to questions with IS response options. Using a research design that combines the advantages of a random assignment between-subjects experiment and the multitrait-multimethod approach in the context of representative sample surveys, we found that responses to A/D rating scale questions indeed had much lower quality than responses to comparable questions offering IS response options. These results attest to the superiority of questions with IS response options.AUTHOR ABSTRACT
The most serious problem facing the United States, according to many scientific and political leaders, is the threat of nuclear war. Yet the standard survey question on the most important problem facing the country has often shown little public concurrence with this assumption. Our article uses experimentation in national samples to test whether this difference can be traced to limitations in either the form or the wording of the standard question. The results indicate that there are some important systematic differences between open and closed versions of the question, and also differences that result from reference to the nation as distinct from the world, but neither type of difference accounts for the infrequent mention of nuclear war on the standard question. Instead, other evidence indicates that most Americans believe that nuclear war is not going to happen at all, or that if it does happen it will be too far in the distant future to be of pressing concern to them personally.AUTHOR INTRODUCTION
Thinking can feel easy or difficult. But what effect does the ease or difficulty of reading a text have on information processing? Can something as seemingly irrelevant as the print font in which information is presented influence how information is evaluated, or even whether it is accepted as true or false? What are the practical implications for everyday life?
KEY QUESTIONS: What is the likely role of metacognitive feelings of ease and difficulty in your own field of psychology? What are the implications for teaching, counselling, advertising, health education, and political communication? What do these influences imply for the rationality of human judgement?