The nonpartisan Urban Institute publishes studies, reports, and books on timely topics worthy of public consideration. The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders.
Note: This report is available in its entirety in the Portable Document Format (PDF).
Chapter 1
Overview of Data Editing and Item Imputation
1.1 Introduction
As discussed in the earlier editing and imputation report on the 1997 NSAF (Report No. 10 in the 1997 series), the National Survey of America's Families (NSAF) was a first-ever effort, put together quickly to respond to major changes in welfare policy at the federal and state levels. The Urban Institute and Child Trends, who jointly led the effort, combined their expertise with the efforts of Westat, an experienced survey firm. Together, the ingredients were present for a successful endeavor. Still, there were many (mostly expected) difficulties, along with some unexpected developments.
By the second (1999) round of NSAF, many of the survey-data-handling problems encountered in the first round had been "solved." We were able to repeat, therefore, our (often hard won) approaches, making refinements where possible and documenting what we did more thoroughly. The present report benefits from this growing experience in two ways. First, much of what we did is not new and we, therefore, will repeatedly refer the reader back to the parallel 1997 report. On the other hand, users of the 1997 round of NSAF may find profit here, too, because we consciously tried to keep consistency between rounds wherever possible. And some aspects of what was done in the first round, notably the handing of the coding of open-ended responses, or "other specifies," are more thoroughly documented here. Because the two rounds were meant to have consistent treatment, we are, in effect, continuing to document the 1997 round.
The present report describes what was done to edit the 1999 NSAF data (see chapter 2), to code written-in entries and match the two rounds of the survey (chapter 3), and to impute for missing item information in 1999 (chapter 4). The public-use file documentation for both rounds, of course, adds many important details on individual variables. In the 1997 report on data editing and imputation, we also had a chapter on general analytic implications of the data flaws detected and undetected. Although referenced here, the same ground has not been repeated, except for a few remarks at the end of the present chapter.
1.2 Data Editing and Data Coding (Chapters 2 and 3)
As detailed in chapters 2 and 3, the data editing process for the 1999 NSAF consisted of three main tasks: handling problem cases, reading and using interviewer comments to make data updates (chapter 2), and coding questions with text strings (chapter 3). Extensive quality control procedures were implemented to ensure accurate data editing.
For the second round of the survey, the work easily separated organizationally into the two activities (editing and coding). Westat did most of the data editing that involved interviewer comments and updates. On the other hand, except for the coding done at the Census Bureau of industry and occupation, all the postsurvey editing and assignment of codes for open-ended questions was done at the Urban Institute.
Strategies for quickly handling data collection problems were developed by Westat during Round 1 of the NSAF in 1997. This development work made the 1999 survey problem resolution much less time-consuming than it had been during Round 1. The volume of problems in Round 2 was not as high as it had been in Round 1, and the problems were not as complex; hence, they could be resolved more quickly. Better CATI (computer-assisted telephone interviewing) software and an improved questionnaire were the major factors that made the process less difficult.
Interviewing for the NSAF was conducted at seven different interviewing facilities that were on slightly different schedules for the beginning of data collection. All comments were read during the first two weeks of data collection at each facility. After this period, only 10 percent of the comments were read.
Limited updates were made to interviews based on comments and problem sheets. The purpose of comment review was changed in Round 2 from searching for data updates to alerting the project team of interviewer training weaknesses. Some comments were used when making updates to completed cases identified through other means, such as programmatic checks run by the Urban Institute. If comments were present when interim problem cases were reviewed, these were taken into account in decision-making.
During Round 1, but not Round 2, Westat conducted coding for "other specify" and open-ended questions. "Other specify" questions were those in which a question had some specific answer categories but also allowed text to be typed into an "other" category. Open-ended questions had no precoded answer categories.
Westat and the Urban Institute had developed an interactive process for defining these codes during Round 1. It was this structure that was built upon in the 1999 survey. Its implementation was done solely at the Urban Institute in Round 2. Often, for "other-specify," we were able to start with the exact decisions made in Round 1 for a respondent comment. Matching these by computer, we were able to ensure complete consistency.
Because data editing resulted in updates to the data, careful quality control procedures were implemented, at both Westat and the Urban Institute. These measures involved limiting the number of staff who made updates, using flowcharts to diagram complex questionnaire sections, consulting frequently, carefully checking updates, and conducting computer checks for inconsistencies or illogical patterns in the data.
1.3 Item Imputation(Chapter 4)
For most NSAF questions, item nonresponse rates were very low (often less than 1 percent), and seldom did we impute for missing responses. The pattern and amount of missingness from round to round varied very little and our approach was virtually identical. The answers to opinion questions were not imputed for any of the cases. The number of missing entries is available on the public-use files, variable by variable, for most items (see chapter 4 of the parallel 1997 NSAF Report on Data Editing and Imputation).
Still, there were important questions for which missing NSAF responses were imputed to provide a complete set of data for certain analyses. For example, the determination of poverty status is crucial, but often at least one of the income items that had to be obtained to make this determination was not answered.
As is the case in many household surveys, NSAF encountered significant levels of item nonresponse for questions regarding sensitive information such as income, mortgage amounts, health care decisions, and so forth. In fact, the income-item nonresponse could range up to 20 or even 30 percent in the NSAF. Hence, the problem could not be ignored. For an introduction to this literature, see especially Kalton (1983), Little and Rubin (1987), Lyberg and Kaspryzk (1997). The three volumes of Incomplete Data in Sample Surveys (Madow et al., 1983) are still, in many ways, definitive.
As will be explained further in chapter 4, the imputation of missing responses is intended to meet two goals. First, it makes the data easier to use. For example, the imputation of missing income responses permits the calculation of total family income (and poverty) measures for all sample families--a requirement to facilitate the derivation of estimates at the state and national levels. Second, imputation helps adjust for biases that may result from differences between persons who responded and those who did not.
The approach used to make the imputations for missing responses in the NSAF was "hot deck" imputation (e.g., Ford 1983). In a hot deck imputation, the value reported by a respondent for a particular question is given or donated to a "similar" person who failed to respond to that question. The hot deck approach to imputing missing values is the most common method used to assign values for missing responses in large-scale household surveys. For the NSAF, a hierarchical statistical matching hot deck design was used (Coder, 1999).
The first step in this imputation process was the separation of the sample into two groups: those who provided a valid response to an item and those who did not. Next, a number of matching "keys" were derived, based on information available for both respondents and nonrespondents. These matching keys vary according to the amount and detail of information used. One matching key represents the "highest" or most desirable match and is typically made up of the most detailed information. Another matching key is defined to be the "lowest" or least desirable. Additional matching keys are defined to fall somewhere between these two; when combined, these keys make up the matching hierarchy.
The matching of respondents and nonrespondents for each item is undertaken based on each matching key. This process begins at the highest (most detailed) level and proceeds downward until all nonrespondents have been matched to a respondent. The response provided by the donor matching at the best (highest) level is assigned or donated to the nonrespondent. For the most part, respondents are chosen from the "pool of donors" without replacement. However, under some circumstances, the same respondent may be used to donate responses to more than one nonrespondent. By design, multiple use of donors is kept to a minimum. An imputation "flag" is provided for each variable handled by the imputation system. In fact, all imputations assigned can be easily tied to the donor through the Donor ID number, because it is retained. The linkages between donor and donee were all kept as part of the complete audit trail maintained throughout the process, although they are not currently being made available on the NSAF Public Use Files.
1.4 Analytic Concerns and Implications
The NSAF, by its very nature, was long and probing—often asking questions never brought together before in the same instrument. Clearly, the length and complexity of the questionnaire contributed to the challenges to be faced--notably in making the data editing more difficult and the level of item nonresponse greater.
Noninterviews have been extensively covered in Reports No. 7 and 8 in the 1997 Methodology Series, plus in selected papers found in Report No. 16 of the 1997 series and Report No. 7 in the 1999 series. Despite the fact that the unit nonresponse was sizable and raised the cost of the survey, there is little evidence of any serious overall bias after adjustment.
Although the path through the evidence is different, we believe that the problems of data editing and item nonresponse are similar in their effect. Their existence in NSAF created extra work, but after the adjustments described here, there is little evidence to suggest significant residual biases beyond those normal to household surveys. This point has already been made in Methodology Report No. 1 of the 1997 series and is elaborated on further in Report No. 15 of that series and Report No. 6 of the 1999 series. In both these reports, NSAF is compared directly with the Bureau of the Census' March Current Population Survey (CPS) and other major national samples.
Despite the high quality of the data editing and imputation in the NSAF, researchers still need to be concerned with how they handle the resulting data. Hot deck imputation, for example, increases the sampling error (Scheuren, 1976) in ways that are hard to measure without special procedures, such as multiple imputation (Rubin, 1987). We recommend that analysts of NSAF data, that have been partially imputed, use an adjustment to their standard errors that corrects for the fact that they do not have as many cases as the variance estimation software (e.g., Wesvar) assumes. One conservative approach to this problem is covered in chapter 4, section 4.6, of this report. References to other options are also provided.
Misclassification concerns arise in any editing or imputation procedure unless the method of assignment perfectly places each missing or misreported case into the right group. The final analyst might employ a reweighting (or reimputation) option rather than just using the imputations provided. To support this option on the NSAF Public Use Files, we have provided a great deal of diagnostic information including the imputation flags, replicate weights, sampling variables, and some variables associated with the interview process itself (e.g., date of interview, number calls to complete ).
As with any data set, researchers will need to be aware of possible anomalies. We believe, however, that these are rare in the NSAF Public Use Files, whether for 1997 or 1999. Still, it is unlikely that we have been able to anticipate all the ways in which the data will be used. Almost certainly there are errors that will be found when researchers carry out their detailed investigations. We would greatly appreciate being informed of any such discrepancies, so they can be brought to the attention of others. Furthermore, depending on the nature of this information, new data sets may be made available.
To give a context to the NSAF effort, there is a bibliography at the end of this report that includes some useful theoretical references plus citations related to the editing and imputation practices used in similar surveys (see especially the Federal Committee on Statistics Methodology Report on Data Editing, Report No 24).
Note: This report is available in its entirety in the Portable Document Format (PDF).
Latest Research
Browse by Topic:
PDF
Printer-friendly summary
