The Uses of Micro-data

The nonpartisan Urban Institute publishes studies, reports, and books on timely topics worthy of public consideration. The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders.

Note: This report is available in its entirety in the Portable Document Format (PDF).

Introduction

The mission of national statistical institutes is to collect and disseminate data. Decades ago, this meant producing books and reports primarily consisting of tabular data — designed to answer pre-defined questions. The increasing complexity of 21st century society, however, has put increasing pressure on such institutes to produce micro-data — designed to allow policy analysts and researchers to pose and answer questions of their own choosing. This pressure creates both opportunity and challenge. On the one hand, the relevance and stature of statistical agencies can be enhanced by their dissemination of data that policy makers can use to answer complex questions quickly. On the other hand, the well-known confidentiality challenges to the creation of public use files and other access modalities have been increased as a result of the development of new types of micro-data, as well as substantial computing and technological advances.

Finding creative ways to address the fundamental tension between data dissemination and the protection of respondent confidentiality goes to the core of each statistical institute mission. Failure to do so has tremendous costs to society. An example might serve to illustrate the point. I have worked with the World Bank on and off for over a decade, in a number of less developed countries. One common characteristic of the statistical institutes of the countries in which I worked was a reluctance to provide access to micro-data — and in every case, this led to incomplete analysis and wasted resources in countries that could afford them least. In one case, the country in question was concerned about the low labor force participation rate of women — which had hampered development for over a decade. Several policy options were on the table — including providing free child care, flexible work-weeks, and subsidized education. However, no micro-data analysis had been undertaken, since although labor force surveys were regularly fielded, they were not even released to the Ministry of Human Resources or the Ministry of Education. We analysed the micro-data and found that, even after controlling for education, industry and occupation, women were paid 60% less than were men — and had been for the decade in question. Our conclusion, which would have been apparent to any analyst working with these data, was that the country in question would be best served by investigating the sources of these earnings differentials, rather than investing in the expensive set of options initially identified. Had the country in question permitted broader access to the micro-data a decade earlier, the appropriate policies could have been in place much earlier.

This is not news to any of you. Indeed, Eurostat has recently issued a new regulation (831/2002) to codify access to confidential data¹. What I would like to discuss is how can statistical agencies determine the "optimal" amount of micro-data to release — and find creative ways to increase this optimum? As an economist, my answer is that an accurate assessment depends on the benefits derived from the use of such data, the costs, and the tradeoff between the two. My goal in this paper is to attempt to explicitly delineate these benefits and costs, identify new changes and summarize the consequences and opportunities for statistical agencies.

Note: This report is available in its entirety in the Portable Document Format (PDF).

The nonpartisan Urban Institute publishes studies, reports, and books on timely topics worthy of public consideration. The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders.

Usage, posting and reprint of materials on the UI web site:

Most publications may be downloaded free of charge from the web site in PDF format. This information may be used and copies made for research, academic, policy or other non-commercial purposes. Proper attribution is required.

Copyright of the written materials contained within the Urban Institute website is owned or controlled by the Urban Institute. Posting UI research papers on other websites is permitted subject to prior approval from the Urban Institute—contact paffairs@urban.org.

If you are unable to access or print the PDF document please contact us or call the Publications Office at (202) 261-5687.