As pay for success (PFS) grows, communities across the nation are exploring how to invest in the most appropriate interventions for this new financing mechanism. To help navigate the challenges of building PFS projects, Urban’s Pay for Success Initiative recently hosted a webinar on how to select and strengthen these interventions, with a focus on why evidence should be a primary consideration.

Evidence not only demonstrates a program’s impact, but also benefits future projects by delivering clear results on what worked and what didn’t. Evidence takes time and effort to measure, and is only as good as the evaluation that produces it. We rank interventions in four categories based on the strength of prior evaluations and demonstrable positive outcomes.

• Strong interventions have undergone enough randomized control trials (RCTs) to conduct a meta-analysis.
• Model interventions have multiple high-quality RCTs or quasi-experimental designs.
• Promising interventions have at least one RCT.
• Potential interventions have yet to be subject to an experimental evaluation, but have a clear link to defined outcomes and an underlying theory of change.

Very few programs meet the “strong” criteria, while the vast majority of existing interventions have no evidence base at all.

Stakeholders can use these guidelines to determine which interventions have the greatest chance of success based on the strength of their evidence base. They should prioritize interventions near the top of the pyramid.

Critical qualities that make interventions “PFS ready”

What else makes an intervention “PFS ready?" Once it is clear there is a strong evidence base supporting an intervention, a PFS-ready program needs to include a focus on prevention, a focus on underserved populations, and the ability to be evaluated rigorously and objectively. When interventions meet these criteria, they are more likely to incur savings to government, prevent negative social outcomes, and demonstrate impact.

Because preventive programs typically have high upfront costs and benefits that accrue far in the future, it can be difficult for governments and agencies to find resources and political will to fund such interventions. PFS mitigates these problems by providing upfront funding for preventive programs and generating savings to government by avoiding would-be expenditures on negative outcomes.

PFS also works best when it targets underserved populations. These populations are often high cost to government because they generate expenditures in multiple systems, creating a need for upfront capital to fill gaps in existing government programming. PFS can insulate governments from potential political fallout because they only pay if the program is successful.

What makes a PFS project so unique is its focus on rigorous evaluation , which benefits both the underserved population and future projects that target similar populations. The first step to meeting this standard is to construct a strong empirical research design. Second, a project needs to collect data in a way that is suitable for analysis. Third, there need to be enough people receiving services to determine statistically significant outcomes. Applying sufficient data to a strong research design will ensure the project makes a contribution to the PFS field, even if it fails.

Looking to the future

Each PFS project has an opportunity to influence future projects, which is why selecting the best available intervention is so crucial to the success or failure of the financing mechanism. A good program cannot be replicated, nor a failed program avoided, without strong evidence and evaluation. By selecting a preventive intervention with high-quality evidence, a rigorous evaluation design, and a focus on an underserved population, stakeholders can be confident their project will be built for success.

The Urban Institute, with a grant from the Laura and John Arnold Foundation, has joined broader efforts to improve stakeholders’ understanding of PFS and to support the creation of better PFS projects at the state and local levels through our Pay for Success Initiative (PFSI). The initiative will support transactions through rigorous research and evaluation design, develop toolkits and templates for the field, provide training and technical assistance, and share lessons learned.

As part of this work, PFSI will be releasing a number of support materials in early 2016, including a guide to the role of evidence in PFS projects, an overview of randomized control trials and their application to PFS, and a description of other evaluation methodologies stakeholders may encounter.

Our criteria for both Model and Promising interventions is informed by the Blueprints for Healthy Youth Development criteria.