PART I: Conceptual Foundations
Chapter 1: Introduction
It is almost tautological to say that policy research is intended to improve decisions made in the public policy arena. It is not at all obvious, however, that this mission is actually accomplished. Indeed, those conducting policy research often wonder if their efforts are used by policymakers in the manner intended, or used at all.
In this book, we examine the connection between a very specific form of policy research—randomized social experiments—and the policy process. Social experimentation has been part of the public policy landscape since the 1960s when Heather Ross, for her doctoral dissertation research, proposed and helped implement the New Jersey Income Maintenance Experiment with funding from the Office of Economic Opportunity.1 Since then, more than 200 randomized field experiments, which have probably cost more than $1 billion (Greenberg and Robins 1986) to conduct and evaluate, have been initiated as pilots to test various proposed changes in social programs. Moreover, use of this evaluation technique, usually referred to as "social experimentation," has accelerated in recent years (Greenberg, Shroder, and Onstott 1999). Although varying greatly in size, budget, focus, and area of interest, social experiments can be characterized as incorporating four common components (Greenberg and Shroder 1997, 4):
- Individuals, families, or organizations are randomly assigned to treatment and control groups. The groups should then differ from each other by chance alone.
- Members of each of the randomly assigned groups receive distinct treatments. The random assignment process, rather than the participants or program staff, determines which treatment each individual, family, or organization receives. Treatment group members face changes in economic incentives, opportunities, or constraints, while members of the control group face no change in the status quo.
- Data are obtained on outcomes of interest—for example, welfare payments, Medicaid utilization, receipt of training and employment services, unemployment insurance receipt, earnings, and health status.
- Comparisons are made of outcomes of interest across the randomly assigned groups, using techniques of statistical inference.
This characterization of "social experiments" is deliberately narrow, as it excludes numerous evaluations of social programs that did not use random assignment, randomized trials that do not seek to measure market or fiscal outcomes (e.g., tests of alternative courtroom or police procedures or clinical tests of health care or teaching techniques), or trials that do not test policy interventions (e.g., fair housing audits or experiments conducted in laboratories by social scientists to investigate behavioral responses to various stimuli). Thus, the emphasis in this book is on field tests of social policies that change economic incentives or constraints and that are evaluated by random assignment and (at least partially) in terms of effects on economic outcomes. Sometimes these field tests are called "economic experiments," although we will use the term "social experiments" throughout this book to emphasize their focus on social policies that are evaluated through random assignment.
During the period from the late 1960s through the 1990s, social experiments ranged from large, lengthy, and costly projects (e.g., the income maintenance experiments) to more modest demonstrations (e.g., job-finding clubs) (Greenberg and Shroder 1997). Whereas many of the experiments focused on examining social welfare policies targeted to low-income populations, some tested programs that addressed problems in such areas as criminal justice, substance abuse, mental impairment, and health insurance. Begun in an atmosphere that was optimistic about the ability of social science research to enlighten public policy deliberations and improve policy decisions, the experiments were justified as an appropriate means for gathering information needed to accomplish these ends.
Objectives of the Research
Understanding the connection between social experiments and the policy process is critical for several reasons. For one, millions of dollars have been spent on them and continue to be spent. The justification made for funding social experiments is almost always an anticipated improvement in policymaking. Social experimentation has become a major, if not the most important, method for evaluating proposed changes in many areas of public policy. Therefore it seems appropriate to ask whether its payoff is worth the substantial efforts it requires. At the same time, to judge the success of social experiments properly, it is important that policymakers, funders of social experiments, the public, and those who conduct social experiments establish appropriate and realistic standards and expectations. The absence of such standards and expectations is likely to lead either to cynicism or to investment on the basis of blind faith. Understanding the connection between social experiments and the policy process can contribute to the establishment of realistic standards and expectations for judging the social gains from experiments. Finally, understanding this connection may provide important insights into ways to increase the social gains from experimentation.
We examine three specific aspects of the connection between social experiments and the policy process. (1) What are the reasons for conducting the social experiments? Our goal here is to understand and describe the process leading to the decision whether or not to conduct social experiments. What do those involved in initiating social experiments hope to accomplish? Why is experimentation rather than other means (for example, case studies, statistical inferences using multiple regression, or simulation) used to accomplish these ends? (2) What is the impact of findings from experiments on subsequent policy deliberation? In what ways, if any, do they exert an influence on policy choices? (3) What factors account for variations in the types and amount of influence that social experiments exert on policy choices?
For each of these questions, there are a variety of possible answers. One set of such answers might be termed the "classical" view. The classical explanation of the decision to conduct a social experiment is an essentially rational paradigm, in which the expected value of the information provided by a social experiment is computed, and a social experiment is conducted if and only if this expected value exceeds the expected costs of conducting that experiment. The classical view of the way social experiments influence the policy process is centered on what we and others term "concrete-substantive use." By this we mean that research findings directly and decisively influence specific policy decisions, thereby resulting in policies being adopted that otherwise would not be. The picture here is of a key decisionmaker, say a cabinet secretary or legislator, reaching a decision largely on the basis of the results of a social experiment. Finally, in the classical view, variations in the types and amount of influence that social experiments exert on policy choices are purely a function of scientific quality.
The classical view, at least at first blush, is very appealing. It is consistent with Progressive ideas of good government. Unfortunately, the classical view is rarely, if ever, consistent with empirical reality. For example, it is difficult to meet in practice the conditions necessary to compute a priori the expected value of the information provided by a social experiment. There is also a wide range of influences on policy choices—some of which are widely viewed as legitimate, others not. Hence, "concrete-substantive use" is both rare and perhaps less desirable than it might be initially viewed as being. Rather, a variety of different types of uses, which are more subtle and less direct than concrete-substantive use, also occur. Similarly, a variety of factors affect the manner and extent to which a given social experiment influences policy choices. Some of the factors pertain to the design and conduct of the experiment, as well as the ways in which its results are communicated, whereas others pertain to the characteristics of the policy environment into which results of experiments are inserted.
In part due to recognition of the limitations to the classical view, several alternative theories and hypotheses have arisen on the connection between social experiments and public policy. These are found particularly in the knowledge utilization and the decision theory literatures. In this book we integrate the theories and hypotheses into a unified, comprehensive framework. That framework is then used as a lens through which to examine the relationship between a number of different previously conducted social experiments and the policy process. Moreover, based on our empirical data, we assess the adequacy of competing characterizations of the relationship between social experiments and the policy process and suggest some directions for further refining the hypotheses and concepts that are embedded within our conceptual framework.
Our Research Approach
Although social experimentation has now become commonplace, relatively little systematic inquiry into the relationship between social experiments and the policy process has been undertaken. Hence, knowledge about the connection between social experiments (or, more generally, policy research) on the one hand and public policymaking on the other remains elusive. In this book, we aim to fill the gap. Our primary mechanism for doing so is five in-depth case studies, each of which focuses on a social experiment or a group of closely related social experiments. Two of these (the RAND Health Insurance Experiment and the Nursing Home Incentive Reimbursement Experiment) focus on an area of great public concern: the quality and cost of health care. The other three (the Income Maintenance Experiments, the Unemployment Insurance Bonus Experiments, and the Welfare-to-Work Experiments) are from the field of income security, the policy area that has been subjected most frequently to social experimentation. All five case studies involve randomized field tests of social policy that meet the criteria listed earlier for a social or economic experiment.
The experiments included in the case studies were run within a specific city or county or several cities or counties, rather than conducted throughout the entire state or country. However, they tested policies that are under the jurisdiction of the federal government and (in the case of the nursing home, unemployment bonus, and welfare-to-work experiments) state governments, but not local governments. Thus, we focus on the use of lessons from these experiments at the federal and state levels, but not at the local level.
The experiments for four of the case studies were selected because they are among the most prominent social experiments ever conducted. As documented in the case studies, they were large, highly publicized, and conducted in multiple sites located throughout the country. Consequently, they were well known to both the research and policy communities. A priori, it seemed to us that the probability that findings from these experiments would be used in the policy process was substantially higher than that for most other social experiments. Thus, it seemed likely that if evidence existed that conducting social experiments is useful, it would be found among these experiments. For contrast, we also selected one experiment that was small, limited to a single site, and known to relatively few individuals—the Nursing Home Incentive Reimbursement Experiment. Many social experiments have been conducted that are similar to the Nursing Home Incentive Reimbursement Experiment in these respects. Our expectation was that little use is likely to be made of the results of such experiments.
Conceptual Framework for the Research
The case studies of these experiments use a common conceptual framework adopted from the knowledge utilization and decision theory literatures. Nearly all previous empirical studies contained in the knowledge utilization literature are either case studies of individual pieces of policy research or surveys that look at policy research in general. The case studies are typically limited in their ability to draw generalizable conclusions, because each case study examines only one policy study. Because the policy studies examined in different case studies tend to differ greatly from one another, the case studies fail to build upon one another. In the surveys, respondents are commonly asked if research findings have influenced policy decisions they have made, but no particular set of research findings or policy issues is specified. Consequently, because of the overgenerality of the questions they pose to respondents, the surveys assume a degree of homogeneity that does not exist among research and policy issues. Because the case studies presented in this book focus on a specific type of research—social experimentation—in which individual studies are based on a core of common elements, they partly overcome these shortcomings. In particular, by exploiting the fact that there is considerable homogeneity across social experiments, multiple case studies, each focusing on different experiments, can increase the generalizability of the conclusions drawn.
The knowledge utilization and decision theory literatures offer explanations and conceptual frameworks for the different ways in which knowledge is used in the policymaking process. These explanations and frameworks have typically been developed by looking at policy research as a general topic, rather than by looking at specific policies or specific pieces of research. Although such generalizations are provocative and intriguing, there has been little systematic effort to test their validity. This study specifically uses the five case studies of social experiments as empirical data to test models of research use.
The cases, two health-related and three from the field of income security, are sufficiently varied that they increase the generalizability of the conclusions drawn from them. At the same time, each is specific enough that key players in the policymaking and research processes in these two policy areas can be asked to focus on the use of specific experimental results, rather than on policy research in general.
The case studies consist of in-depth analyses of the process surrounding each of the social experiments that are examined:
- Why the experiments were undertaken. The origins of the experiments, decisions that led to running them, and environmental context in which the decisions were made.
- The range of use of findings from the experiments. In part, that was determined by key players such as researchers involved in the experiments, policymakers involved in the debate about the issues addressed by the experiments and in subsequent implementation of policies that potentially could have been influenced by the experimental results, and social thinkers whose views might have been influenced by the experiments.
- Factors influencing whether and how the experiments were used. Such factors include the characteristics of the experiments themselves and the policy environment after findings from the experiments were available.
Organization of This Book
This book begins by providing background information on social experimentation. Thus, chapter 2 first presents an overview of the history of social experiments since they began in the late 1960s and then examines trends in social experimentation over time. Chapter 3 discusses the relevant knowledge utilization and the decisionmaking literature and describes a conceptual framework that is drawn from that literature and used in conducting the case studies. The chapter also describes the methodology used in collecting information for the five case studies.
Chapters 4 through 8 present the five case studies themselves. Relying on a common conceptual framework, each of these chapters examines the origins of the experiment or set of experiments covered by the chapter; describes the experiment(s); summarizes key findings from the experiment(s); and, most important for this work, uses the conceptual framework developed in chapter 3 to examine the effects of the experiment(s) on policy.
Chapter 9 augments the five case studies by examining the dissemination and use of findings from three recent social experiments, each of which tested a welfare-to-work program in a different state. The chapter examines whether state welfare officials learn about findings from experiments conducted in other states and, if they do, how they learn of the findings and whether and how they use that information for policymaking in their own states.
Finally, we end (chapter 10) with inferences and conclusions that are drawn across the case studies. Specifically, we assess whether lessons from the case studies can be applied to other situations and circumstances. We focus particularly on the implications of the case studies for circumstances in which the payoff from conducting social experiments is greatest, the utility of experiments for policy purposes, and means of enhancing their usefulness. We also draw some conclusions about whether the knowledge utilization and decision theory literatures provide a useful conceptual framework for examining the origins of social experiments and their role in the policy process.
NOTE
1. Telephone interview conducted with Heather Ross on December 2, 1994.
