Research Report Ten Key Design Elements for Rigorous Impact Evaluations in Child Welfare
Subtitle
A Desk Reference for Evaluators
Katrina Brewsaugh, Sarah Prendergast
Display Date
File
File
Download Report
(1.88 MB)

Rigorous impact evaluations tell us whether child welfare programs help families and children—in other words, whether they are effective. Impact evaluations that show a program works can lead to increased support from funders or wider program use as agencies want to use effective programs. Child welfare agencies and service providers need to produce impact evaluations that use accepted best practices for study design and execution.

This desk reference summarizes best practices for 10 key elements when designing a rigorous impact evaluation in child welfare settings. The purpose of this desk reference is to help evaluators design rigorous impact evaluations that can help show a program works and, for each design element, understand the criteria for inclusion across six clearinghouses relevant to child welfare programs.

Key Findings and Highlights

We summarize the following 10 design elements that are often mentioned by researchers, legislation, and clearinghouses when they talk about rigorous evidence that something works:

  1. choose an RCT or QED study design
  2. determine the sample size needed
  3. create a process to assign individuals
  4. create a process to assign clusters
  5. create fidelity measures
  6. select outcomes and instruments
  7. create a process to gather baseline data
  8. create a process to collect outcome data
  9. create your analytic plan
  10. preregister your study

Recommendations

Child welfare program evaluators should follow the best practices for each design element discussed in this desk reference when designing a rigorous impact evaluation. However, this reference cannot cover every important aspect of designing and executing an impact evaluation in child welfare, nor all the various clearinghouse requirements. We recommend that agencies looking to evaluate child welfare programs speak with methodological experts about their specific context. Evaluators or agencies that want a program to be rated by a specific clearinghouse should review that clearinghouse’s specific design and execution standards.

This resource is part of the Roadmaps to Building Evidence in Child Welfare series—a collection of instructional resources about conducting child welfare evaluations. You can find more practical guidance on data, evaluation, and evidence in the child welfare field from CWEST in materials from the Child Welfare Evidence Building Academy, a program of trainings for child welfare agency staff, practitioners, and evaluators.

Research Areas Child welfare
Policy Centers Center on Labor, Human Services, and Population
Research Methods Research methods and data analytics Performance measurement and management