Improving State Policy Evaluation

Brief

Improving State Policy Evaluation

Using the Synthetic Control Method

Abstract

This fact sheet explains how to use the synthetic control method for evaluating state policy.

Full Publication

Randomized controlled trials (RCTs) are the gold standard for evaluation across fields ranging from medicine to social policy. True RCTs, however, are rarely possible in state policy evaluation because policies are not implemented randomly across states, and states are not always comparable on factors that can influence policy outcomes, such as demographics or economics. The synthetic control method is an evaluation tool that policymakers can use to overcome some of these challenges. It is ideal for examining a policy unique to a particular state or region, from nuclear power facility placement to tourism development, and can complement more qualitative case studies.1

What Is the Synthetic Control Method?

The synthetic control method analyzes changes in state public policies by comparing the outcomes of two versions of a state. One version is the actual state that has undergone the policy change. The other version is artificial, created to estimate the outcome as if the policy was not enacted. Based on variables that predict the outcome before the policy change, this artificial state is created from a weighted combination of other states, or “donor states,” and matches the actual state’s outcome path before the policy is enacted. The difference in outcomes between the affected state and its artificial (synthetic control) counterpart after enactment is intended to reveal the policy’s effectiveness.

How to Use the Synthetic Control Method

The researchers who popularized this method have made their software package, Synth, publicly available, providing an open-source tool for policymakers to do synthetic control analysis. The following guidelines give practitioners a place to start when using the method. After determining the policy and period of interest, the steps to generate a synthetic control state are as follows:

Step 1: Identify predictor variables that you expect to affect the outcome.

  • Choose factors that should affect outcomes in the state both before and after the policy is enacted, such as income, race or ethnicity, age, or other demographic, economic, or policy variables.
  • Select the range of years preceding policy enactment over which these variables will be averaged. Usually, the longer the better, and variables can be used even if some years are unavailable.
  • Include several values of the outcome variable from the period before the policy change (known as “outcome lags”). Choose values that highlight the trend of the outcome before treatment, but do not include all lagged values of the outcome variable in your model.

Step 2: Identify possible donor states to create the synthetic control state.

  • Exclude any states that enacted similar policies during the selected period. Relatively small policies may be acceptable.
  • The values of these states’ predictive factors should be comparable to those of the state being studied before treatment. Ideally, each predictor will have potential donor states with values greater and less than the value in the state with the policy change.

Step 3: Choose a method for selecting the predictor variable weights.

  • The Synth software generates the predictor weights. The standard method used in the Synth software minimizes the outcome’s mean square prediction error over the full pretreatment period.

Step 4: Examine whether the control state that you have generated is a good match for the state you are evaluating.

  • Evaluate how closely the outcome path of the control state follows that of the treated state based on appearance and the difference between predicted and observed values (using the root mean squared prediction error).
  • If the fit appears poor, test it with a model that includes outcome lags for all years before policy enactment. If the control state poorly matches the treated state, the synthetic control method should not be used.
  • Examine the state weights generated by the Synth software to judge the similarities between the donor states and the treated state. It can be more important for outcomes of donor states to have a trend similar to that of the treated state than for the donor states to have a similar average.
  • Review predictor weights to determine the predictor variables’ strength in explaining the outcome.

Step 5: To evaluate the significance of the results, run the synthetic control method separately on each state in the donor pool, as though it is a treated state, in what is known as a “placebo test.”

  • If the post-policy difference between the treated state and its synthetic control state is larger than the difference for most of the states in the placebo tests, there is evidence that the treatment had an effect.
  • Evidence from the placebo tests should be treated as suggestive of a significant effect rather than definitive.

Step 6: Conduct sensitivity analyses to further test the credibility of the results.

  • Test whether and how much your results change based on different choices, such as selection of outcome lags, predictor variables, year range, and the method for choosing predictor weights.

1 For more information, see Robert McLelland and Sarah Gault, The Synthetic Control Method as a Tool to Understand State Policy, (Washington, DC: Urban Institute, 2017). http://www.urban.org/SyntheticControl. ^

Cross-Center Initiative

Cross-Center Initiative: 
To reuse content from Urban Institute, visit copyright.com, search for the publications, choose from a list of licenses, and complete the transaction.