2002 NSAF Collection of Papers

Report No. 6 in the 2002 NSAF Methodology Series

The nonpartisan Urban Institute publishes studies, reports, and books on timely topics worthy of public consideration. The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders.

Note: This report is available in its entirety in the Portable Document Format (PDF).

1. PAPERS PRESENTED AT THE AMERICAN ASSOCIATION FOR PUBLIC OPINION RESEARCH CONFERENCE

Overview

Chapter 1 consists of six NSAF methodology papers that were presented at the American Association for Public Opinion Research (AAPOR) conference and have not been published elsewhere besides in the conference proceedings. While most of these papers can be found in the AAPOR proceedings, the versions included in this report may vary because of the page size and time deadlines associated with papers submitted for proceedings publication.

All papers in this chapter are based on the 2002 NSAF data collection experience, except for the first paper ("Effects on Survey Estimates from Reducing Nonresponse"). The first paper uses 1999 data and is included in this report because it was presented after the release of the 1999 collection of papers methodology report.

Effects on Survey Estimates from Reducing Nonresponse

Adam Safir, Rebecca Steinbach, Timothy Triplett, and Kevin Wang

1. Introduction

Using a variety of procedures designed to maximize response rates, survey organizations expend sometimes extraordinary efforts to minimize the potential for nonresponse bias. Given that nonresponse bias is a function of both the nonresponse rate and the difference between respondents and nonrespondents, maximizing response rates is a sensible approach to minimizing the potential for bias contributed by less-than-perfect survey participation rates.

The second part of the equation is by definition a more complicated component to address. A number of approaches have been suggested for measuring the size of the difference between respondents and nonrespondents, despite the unobserved status of the latter. One such method is to use difficult-to-interview respondents, obtained through increased call attempts, higher incentives, or an extended field period, as proxies for nonrespondents.

Although it is assumed that additional efforts to obtain interviews with the difficult-to-interview will improve precision and reduce nonresponse bias (Lynn et al. 2002), when the interviews obtained as a result of these efforts display characteristics similar to interviews already conducted with easier-to-interview respondents, researchers may arrive at one of two conclusions: (1) the difficult-to-interview, or nonrespondents-by-proxy, do not differ in meaningful or systematic ways from other respondents, thus implying ignorable nonresponse; or (2) a core group of nonrespondents remains unmeasured, thus suggesting the potential for nonignorable nonresponse bias. Faced with either prospect, researchers may question the extent to which additional interviewing efforts are merited, given the absence of identifiable nonignorable nonresponse bias (1) or apparent ineffectiveness (2). For example, where there is little indication of a bias reduction resulting from extended efforts to obtain additional interviews, the survey organization may consider a redesign of expensive refusal reworking procedures (Scheuren 2000).

This paper presents the results of research conducted to analyze the effects of efforts to minimize the potential for nonresponse bias in the 1999 round of the National Survey of America's Families (NSAF). In particular, this research was motivated by questions about the efficacy of maximizing response rates on minimizing nonresponse bias.

In the first major analysis component—level of effort—we address the effect that increasing the level of effort expended to increase participation rates has on reducing nonresponse bias by comparing the characteristics of persons in easy-to-interview households to the characteristics of persons in difficult-to-interview households. These groups are defined by number of calls to contact and number of refusals. As the literature has suggested that the characteristics of noncontacts and refusals may differ substantively from each other as well as from the "average" respondent, particular emphasis was given to examining differences between each subset of the difficult-to-interview, the difficult-to-contact (5+ calls to contact) and the reluctant-to-participate (2+ refusals), and the average interviewed household. In addition to comparing these groups within the 1999 survey round, we compare measures associated with varying levels of contactibility and cooperation across survey rounds.

The second major analysis component—potential for nonresponse bias—focuses on assessing the potential for nonresponse bias due to unmeasured sample elements, treating difficult-to-interview observations as informative of the noninterviewed. Within this analysis step, we also report on the results of a comparison of sampling frame data across easy-to-interview, difficult-to-interview, and noninterviewed households, defined by completion status in NSAF and a short follow-up survey. The data were compared across these three groups to assess the appropriateness of using difficult-to-interview respondents as proxies for the noninterviewed.

Note: This report is available in its entirety in the Portable Document Format (PDF).

The nonpartisan Urban Institute publishes studies, reports, and books on timely topics worthy of public consideration. The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders.

Usage, posting and reprint of materials on the UI web site:

Most publications may be downloaded free of charge from the web site in PDF format. This information may be used and copies made for research, academic, policy or other non-commercial purposes. Proper attribution is required.

Copyright of the written materials contained within the Urban Institute website is owned or controlled by the Urban Institute. Posting UI research papers on other websites is permitted subject to prior approval from the Urban Institute—contact paffairs@urban.org.

If you are unable to access or print the PDF document please contact us or call the Publications Office at (202) 261-5687.