urban institute nonprofit social and economic policy research

Impact Evaluation: Regression Discontinuity

by Austin Nichols

What is it used to measure?

Regression discontinuity methods are used to estimate how a treatment or policy change affects individual outcomes in the absence of a randomized controlled trial.

How does it work?

Estimates of the causal effect of a policy change on individual outcomes typically founder on the infeasibility of using an experimental design. In an experiment, participation in a treatment group is randomly assigned so that outcomes may be compared across treated and nontreated groups and cannot be attributed to selection bias (that is, to characteristics treated participants do not share with nontreated participants). Therefore, the average difference in outcomes is an unbiased estimate of an average treatment effect.

But many experiments must remain hypothetical because there is no legally, politically, or economically feasible experiment. For example, our desire to estimate the effect of teacher qualifications on student outcomes does not justify randomly assigning students to different classrooms, across a variety of schools and school districts, for several years. We are compelled in such cases to estimate treatment effects using available data. Methods that identify treatment effects in the absence of an experiment are known as quasi-experimental methods and include instrumental variables and regression discontinuity.

To use a regression discontinuity (RD) design, we must assume participants are assigned to treatment groups based on an observed variable (call this the assignment variable). In theory, participants with values of the assignment variable above a threshold are assigned to a treatment group, and those with values of the assignment variable below the threshold are assigned to a control group (that is, they do not receive the treatment). Thus, treatment and control groups are not assigned randomly; we instead observe the measure underlying their assignment.

The second crucial assumption is that the outcome variable is a continuous and smooth function of the assignment variable, especially near the threshold. There is no meaningful way to test the assumption, but in practice, a good theoretical justification combined with actual data plotted on a graph constitute convincing evidence that the assumption is well-founded.

Suppose the poverty rate in a district is used to determine a school’s eligibility for a specific policy intervention (the treatment). Thus, the poverty rate is the assignment variable, and the minimum poverty rate at which districts are eligible for the intervention is the threshold. The graph below illustrates the method graphically, where a positive effect of eligibility is visible as a jump at the threshold in predicted outcomes, where outcomes are modelled as a function of the poverty rate. Data points should be graphed along with fitted curves, so the plausibility of fitted curves is immediately obvious.

Figure 1

The RD estimate of the treatment effect (the gap at the dotted line in figure 1) is measured by the estimate of the coefficient on a variable indicating values of the assignment variable above the threshold in a regression equation, and standard statistical tests of significance are simple to calculate. Alternative specifications could include higher-order polynomial terms, different distributional assumptions (leading to different regression techniques), or a variety of kernel regression techniques in the place of polynomials (in fact, a local polynomial regression, where only neighboring observations are included at each point, is standard in RD analyses).

To conclude that the RD design is appropriate, other variables for which no theoretical relationship to the observed variable is surmised (for example, the number of students in a school district) would also be plotted versus the assignment variable, along with a comparable set of regression results. If these irrelevant variables show no statistically significant jump at the threshold then the observed effect on the outcome variable is more plausibly traced to the treatment.

RD’s advantage over IV is the relative robustness of results to distributional assumptions and analysis methods. The advantage of IV over RD is the possibility, if assumptions are valid, of extracting the causal impact of treatment on outcomes (whereas RD estimates the effect of eligibility on outcomes).

Research examples

"E-Rate and the Digital Divide: A Preliminary Analysis From the Integrated Studies of Educational Technology"

"The Integrated Studies of Educational Technology: A Formative Evaluation of the E-Rate Program (Draft)"