Tim Triplett, a research methodologist and part of Urban Institute's new Statistical Methods Group, is coauthor of Lessons Learned from the National Survey of America's Families (NSAF). As an NSAF survey manager, he worked closely with survey designers, statisticians, and policy analysts. The NSAF was conducted in three rounds—1997, 1999, 2002—and gathered information each round on more than 40,000 U.S. families, mostly low-income.

Five Questions Archives

1. What are the survey's important contributions?

NSAF's main goal was to identify low-income families and, even today, no other survey has gleaned as much information on this group. The American Community Survey—essentially, the Census' annual long form—doesn't ask the questions on child well-being and adult well-being that NSAF does.

Defining family structure is another important contribution. NSAF not only shows the typical low-income family, but also shows the many different family dynamics that get noted in this survey—children without parents in the home, grandparents raising kids, cohabiting partners, and unmarried biological parents. NSAF also does a good job of tracking information on nonresident mothers and fathers.

The questionnaire content was another important contribution. It was particularly useful in determining how we measure whether people have health insurance. The federal government adopted the NSAF method after we successfully added a verification question to more accurately estimate the uninsured population.

From a design perspective, the embedded testing of incentives and different calling strategies really stood out. Various surveys write letters or offer money to get people to respond. NSAF was large enough that we could try different types of incentives and decide what works best.

2. How did NSAF enhance the policy debate on low-income families?

There's controversy on how government measures poverty. What the NSAF did is novel in that it both replicated the way the federal government measures poverty and defined poverty in its own way by creating what we call a social family.

Government defines poverty very rigidly. We expand that definition to cover same-sex partners or partners who are contributing income. That really enhances the debate because it provides a measurement of poverty that the government doesn't detect.

Also, our well-being measures can be cross-referenced. It helps to be able to see, for instance, how a parent suffering from mental illness might affect how well the child is doing in school. That extra dimension is a big enhancement.

Also, no other household survey queries on child care. When parents go to work, where do the kids go? And, how do low-income parents afford reliable care? Many of our papers based on the NSAF research address those child-care issues. This is an angle that no one else really examined.

3. Why is NSAF important today?

There are still many things people want answered about low-income families during the survey period. Major transformations were occurring in welfare and other social programs in the late 1990s. Just this month we had 150 people newly registered to use the public-use NSAF files.

As a private endeavor, NSAF was designed to be experimental and flexible.  Future household survey designers have a lot to learn from that. Government surveys are only now running up against the participation problems that happened sooner in the private sector. As refusals continue to grow, the government has to consider strategies to deal with them. NSAF's struggle provides lessons to those designers.

We did considerable work on incentives to encourage more survey respondents. Incentives are still a relatively new idea for the federal government and NSAF is one of the few surveys that experimented in this field.

Different ways of collecting survey data—mixed mode—is another area where the NSAF carved new ground. We had two sampling frames: telephone and in-person. But when we interviewed in person, we used cell phones to try to make the two modes more compatible.

As people start mixing modes, compatibility is imperative. That's a key lesson learned that is very important today. The effective interviewer will get a different response on a sensitive question in person compared to on the telephone. A mail survey or over-the-Internet will also produce different responses.

Cutting across many issues also put NSAF out front. Our survey questions were formulated to monitor and analyze the well being of American children and families in cross-cutting terms: health care, income security, social services, and job-training programs. The challenge was to do so without overtaxing survey respondents.

4. What were the advantages of NSAF being a nongovernmental survey?

NSAF's design did not have to go through the rigorous OMB [Office of Management and Budget] survey-review process. OMB reviews questionnaire and survey designs of all government-sponsored surveys to make sure that the sample will be representative and useful. OMB doesn't want federal agencies to unduly burden the public.

The oversight of the OMB process can be very useful. However, the process would have slowed down a survey as large and broad as NSAF, making it harder to experiment within the survey design and to make changes as needed. Some of the changes might have been delayed until they weren't useful any more. Or, potentially, someone at OMB who didn't understand the design might have made the wrong decision.

Studies suggest that when the federal government collects data, people may respond differently. The government seal might trigger inherent bias in how they respond. NSAF was a large enough private survey to provide a good comparison.

We weren't tied to the same rules as government surveys either. For instance, we could offer incentives for people to participate. The federal government is only just now considering incentives.

With the NSAF, we have a closer collaboration between the survey designers and policy makers. Agencies in the government don't work as closely. There's more compartmentalization. People who are designing surveys aren't as closely tied to policy analysts. NSAF had a tight-knit group.

5. What are the lessons for future surveys?

Closer ties between policy analysts and survey designers are essential. A survey isn't like a car that you can leave with mechanics and then drive away at the end of the day.

Research methodology no longer has much shelf life, and survey designers should consider experimental design prior to the project's start. Even on the NSAF, we learned that the strategies we used in 1999 became less effective just three years later.

Changing technology has really changed the way people communicate. Plus, different people communicate in different ways. Some people aren't comfortable answering surveys by telephone anymore. Some people are comfortable with Internet surveys; some people aren't. It's going to take a multi-mode approach to reach a broad public.

Another lesson has been the importance of allowing for population shifts. The changes in the population between the 1990 census and the 2000 census forced us to reweigh the Hispanic population. Also, the population shifted vis-à-vis where the telephone households lived. We need to keep the sample fresh.

Another area is the proxy response. We asked respondents to provide answers about their spouses and then compared them to spouses' actual responses. We found they were pretty reliable. Many people are very difficult to get to participate in surveys. Do we just lose these people from surveys? Or do we allow proxy responses? A lesson from the NSAF is that proxy reporting can be reliable.

Flexibility seems to be at the core of all these lessons. Survey designers and policy makers both are going to have to think about very flexible surveys. It's a very busy society that we're in right now and very hard working. People need to be able to choose how to respond. Also, different incentives work for different populations. We have to be more creative in finding the right match.

Timeliness was an issue, but that's just the name of the game. NSAF processed the information as quick as any survey, but it took time to get the data out to the public. The problem is that it was a nine-month data collection, so if you start early in 2002, you won't finish until late 2002.