Propensity Score Methods

by Bowen Garrett

What are they used to measure?

Propensity score (PS) methods are often used to estimate the impact of a policy by comparing people subject to the policy (the treated group) to people not subject to the policy but as similar to the treated group as possible (the untreated group).

How do they work?

PS methods can be used to estimate the effects of a binary characteristic or policy intervention from observational data when, as is common, using an experimental design is not feasible. PS matching is commonly used to draw a comparison group from the untreated group, such that the comparison group is indistinguishable from the treated group.

A simple PS matching approach is as follows:

1. Pooling treated and untreated groups, assign each subject a PS (the conditional probability that a subject is in the treated group). The PS can be estimated by using a regression of treatment status on the set of explanatory variables and computing each subject’s predicted probability of being treated.

2. Match each subject in the treated group to a subject that has a similar PS in the untreated group. The untreated cases form the comparison group. A common variation is to match to more than one untreated subject with a similar PS. If the observed characteristics of the treated and PS-matched comparison groups are sufficiently similar, move to step 3. If they are not, refine the PS model until similar treated and comparison groups are obtained.

3. Finally, estimate the effect of treatment as the (possibly weighted) difference in means of the outcome measure between the treated and PS-matched comparison groups.

PS matching is an alternative to using more traditional regression analysis to estimate the impact of a policy. But PS and regression methods estimate somewhat different quantities. The coefficient on treatment status in a regression of the outcome measure on treatment status and explanatory variables yields an estimate of the "average treatment effect." The PS matching method yields an estimate of "the effect of treatment on the treated." The two coincide if one assumes that the effect of treatment does not vary across subjects. Thus regression and PS matching yield conceptually and numerically different estimates.

Major benefits of the PS matching approach are its transparency and its ability to force a direct test of the extent that the distribution of characteristics in the treated and untreated groups overlap. This is because the method requires that the distribution of the propensity scores for the treated and untreated groups overlap sufficiently, implying overlap in the distribution of observed characteristics.

Lack of overlap tends to yield imprecise estimates and cast doubt on the ability to use untreated subjects to estimate what outcomes for the treated group would be in the absence of treatment. Regression would yield more precise estimates, but rely more heavily on the functional form assumptions of the regression model. Regression methods do not lend themselves to a direct investigation of such overlap, and without one, potentially suspect comparisons may go undetected.

An alternative to matching is a PS–based reweighting method, whereby untreated observations are reweighted so that the distribution of observed characteristics in the treated and untreated groups are nearly similar. Reweighting bypasses the need to choose from different matching methods and seeks to utilize all observations in the untreated group.

For PS methods to properly estimate the effect of treatment, it must be assumed that no unobserved characteristics are related to both treatment status and the outcome measure, having conditioned on the observed characteristics. Therefore, PS methods alone, like standard regression methods, cannot address omitted variable bias, usually a key concern in impact studies that use observational data.

Research examples

"Low-Income and Low-Skilled Workers' Involvement in Nonstandard Employment Final Report"

"Married and Unmarried Parenthood and Economic Well-Being: A Dynamic Analysis of a Recent Cohort"

"Pathways to Work for Low-Income Workers: The Effect of Work in the Temporary Help Industry"

"Tests of Nonexperimental Methods for Evaluating the Impact of the New Deal for Disabled People" (PDF; link to Department for Work and Pensions web site.)