Report No. 8 in the 2002 NSAF Methodology Series
The nonpartisan Urban Institute publishes studies, reports, and books on timely topics worthy of public consideration. The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders.
Note: This report is available in its entirety in the Portable Document Format (PDF).
1. INTRODUCTION
This report gives the response rates achieved in the 2002 National Survey of America's Families (NSAF). In addition to giving the response rates, this report describes the methods used to compute the rates, compares the 2002 response rates with the rates from previous NSAF data collections, and examines the environment for conducting random digit dialing (RDD).
The 2002 NSAF is the third round of the survey; the two earlier rounds were conducted in 1997 and 1999. The content of the 2002 survey was very similar to that of the earlier NSAF data collections in order to provide valid and effective measures of change over time. Two important design changes in the 2002 survey are the elimination of the nontelephone sample for the study areas and the elimination of Milwaukee as a study area separate from Wisconsin. These and other design changes are described later in this chapter because they have implications for analyzing changes in response rates over time.
The response rates presented in this report are weighted to account for the different probabilities used to select the sample. In this important way, the rates reported here differ from those given in other reports in the 2002 NSAF report series. For example, the response rates in 2002 NSAF In-Person Survey Methods differ from those in this report because of the use of weights. The reasons for using weighted rates are discussed further later in this chapter and more completely in chapter 2.
Chapter 3 gives a brief description of the procedures used in conducting the survey as a means of explaining how the response rates were achieved. These procedures are described in more detail in the two reports in the series covering data collection, 2002 NSAF In-Person Survey Methods (Report No. 5) and 2002 NSAF Telephone Survey Methods (Report No. 9).
Chapter 4 details the methods used to compute the response rates. The methods used are relatively complex and related to the procedures used to weight the data. Chapter 5 gives tables of response rates for the overall national sample and for key subgroups. Chapter 6 examines how the response rates from the 2002 NSAF compare with those achieved in Round 1 and Round 2 and provides some information on changes in other RDD survey response rates.
The rest of this chapter gives an overview of the survey, the reasons for reporting response rates from this survey, and the relationship of this report to others in the NSAF methodology series.
1.1 Overview of the Survey
The NSAF collects information on the economic, health, and social dimensions of the well-being of children, adults under the age of 65, and their families in 13 states and the balance of the nation. The Urban Institute selected these study areas (see figure 1-1) in because they represent a broad range of fiscal capacity, child well-being, and approaches to government programs. Data are also collected in the balance of the nation to permit estimates for the United States as a whole. We briefly outline the sample design here. Complete details are provided in 2002 NSAF Sample Design, Report No. 2.
Figure 1-1.
Study Areas |
|
| Alabama |
Massachusetts |
New Jersey |
|
| California |
Michigan |
New York |
Wisconsin |
| Colorado |
Minnesota |
Texas |
Balance of nation |
| Florida |
Mississippi |
Washington |
|
|
The primary goal of the survey in all three rounds is to obtain social and economic information about children in low-income families (those with incomes below 200 percent of the federal poverty level) since the impact of new federalism was likely to be greatest on these children. Secondary goals include obtaining similar data on children in higher income families, plus adults under age 65 (with and without children).
One major change in the design for the 2002 survey was that Milwaukee County, Wisconsin, which had been a separate study area with its own sample in 1997 and 1999, was no longer a separate study area. The Milwaukee County study area was included with the rest of Wisconsin as a single study area.
Another major change in the sample design was to eliminate the nontelephone sample in the study areas representing specific states. The NSAF uses two frames-an RDD survey of households with telephones and an area sample conducted in person for those households without telephones. In 1997 and 1999, this dual-frame approach (see Waksberg et al. 1997) was used for each study area and for the nation. In 2002, only a national-area sample was selected and the same dual-frame method was used to produce national estimates. No separate sample of households without telephones was selected in the study areas, so estimates for the study areas are based only on the RDD sample selected from each area. The weights of the RDD samples in the study areas were adjusted to reduce the potential bias associated with not sampling nontelephone households.
The national-area sample consisted of a subsample of primary sampling units (PSUs) used in the 1999 survey. The 1999 PSUs were subsampled in the study areas, while all the PSUs from the balance of the nation were retained. Since block groups with very high telephone coverage rates as of the 1990 Census were excluded in Round 1 and Round 2 sampling, this same restriction was carried forward to the 2002 survey of nontelephone households.
Another new feature introduced in the 2002 survey was the subsampling of refusal cases. A random sample of telephone numbers was selected, and only those identified in the subsample were followed to completion if they refused the screening interview. The numbers that were not subsampled were classified separately to make it easier to compute response rates.
In the RDD sample, a screener-based subsampling of households was used to sample low-income households at a higher rate than other households. A very short income question was asked during the RDD screening interview, and those that reported an absence of children or reported incomes above 200 percent of the poverty level were subsampled.
Within both the RDD and area samples, household members were subsampled to reduce the respondent burden. If there were multiple children under age 6, one was randomly selected. The same was done for children 6 to 17 years old. Data were collected from the most knowledgeable adult (MKA) in the household for the sampled child. During the MKA interview, data were also collected about the MKA and about his/her spouse or partner. Most questions asked about the MKA were repeated in reference to the spouse or partner; however, some questions on health insurance and health care usage were asked in reference to only one of the two. The target of these questions was randomly assigned to either the MKA or his/her spouse or partner. Questions asked only about the MKA were those related to feelings, religious activities, and opinions.
Other adults in households with children were subsampled, as were adults in adult-only households. Adults were eligible only if they would not have been the MKA for other children in the household if those children had been selected. Self-response was required for sampled adults-with proxy data collected about his/her spouse or partner (if living in the same household). Data were not collected directly from the spouse of a sampled adult. As in the MKA interview, some questions related to feelings, religious activities, and opinions were asked only about the sampled adult.
1.2 Purpose of Reporting Response Rates
Response rates are used to assess various aspects of a survey's quality (Madow et al. 1983). One purpose of reporting response rates is to provide a measure of how successfully the survey obtained responses from the sample. This outcome is related to the potential for nonresponse bias, and is the main objective of this report. To accomplish this goal, response rates are weighted so that the weighted response rates indicate the proportion of the population that responded. For example, since the balance of the United States includes half of the total population, the response rates are weighted so the balance of the United States accounts for 50 percent of the overall response rate.
The rationale for using weights in computing the response rate is that the bias of an estimate, 
, is a function of the response rate. Suppose the population can be partitioned into strata of respondents (R) and nonrespondents (NR), and the estimate from the survey is computed from the sample from the respondent stratum (without any modification of the inverse of the selection probability weights). The bias of a survey estimate of a mean is
where r is the appropriately weighted response rate and the quantity on the right is the difference in the means between the respondent and nonrespondent strata (Lessler and Kalsbeek 1992). This formulation clearly shows the bias increases as the response rate decreases if the difference in characteristics between respondents and nonrespondents remains constant. If the response rates are not weighted, this relationship does not hold. Returning to the example, if the balance of the United States response rate is not weighted to account for 50 percent of the population, then the response rate cannot be used in the bias equation.
While weighted response rates are relevant for understanding the potential for bias, the most appropriate rates for operational purposes may be unweighted response rates. Unweighted response rates provide information on how many sampled units (households, persons, etc.) participated and are valuable for evaluating the effectiveness of specific field operations. Unweighted rates are used in other reports (such as the Sample Design, In-Person Survey Methods, and Telephone Survey Methods reports) for this purpose. This report presents weighted rates only, and as a result, the rates may not agree with those in other reports in this series.
Response rate reports from the 1997 and 1999 surveys (NSAF Report No. 8 in the series for each year) also included a section on the results of experiments related to the response rates. But this report does not cover any experiments from the 2002 survey. There were a number of experiments in the 2002 survey related to response rates and other operational issues to improve the quality and efficiency of the study. Because of the number of experiments and their varied nature, a separate report on experiments from the 2002 survey is being prepared.
1.3 Summary of Response Rates
The remaining chapters in the report define the terms and methods used to compute response rates for the 2002 NSAF. The key response rates are summarized in this section, but a more complete understanding of the terms and procedures used in defining these rates may require reading the material in the following chapters. The overall national screener response rate for the 2002 NSAF is 66 percent. This response rate essentially describes the percentage of households that participated sufficiently to determine if there were any members of the households sampled for the survey. The 2002 NSAF national child extended response rate is 84 percent. This rate describes the percentage of completed interviews conducted for those children who were sampled. The product of the screener response rate and the child extended rate gives a national joint child response rate of 55 percent. The adult extended and joint response rates are approximately 2 to 5 percentage points lower than the corresponding child rates. The national adult extended response rate is 79 percent, and the adult joint response rate is 52 percent.
More details on the response rates for the screener, child, and adult interviews are given in chapter 5, including response rates for subgroups. This chapter also presents the response rates for the study areas. Chapter 6 compares the response rates for Round 3 with those for the two earlier rounds. The main finding is that the screener response rate decreased substantially over the 5 years, while the extended response rates remained at about the same level for all three rounds. Consequently, the overall response rates that are the product of the screener and extended rates decreased because of the screener rate decline. The context for the decrease in response rates is discussed in some detail in chapter 6.
Note: This report is available in its entirety in the Portable Document Format (PDF).
Latest Research
Browse by Topic:
PDF
Printer-friendly summary
