Impact evaluations tell us how well an intervention for children and families is working. Planning for and conducting an impact evaluation requires specific knowledge, skills, and resources within the child welfare agency. This report outlines what needs to be in place for an impact evaluation to succeed. We hope that child welfare administrators use this information to assemble a strong team capable of undertaking an impact evaluation and producing much needed evidence for what works in child welfare.

Key Findings and Highlights

• Ensuring that the intervention is operating as intended is an important first step in determining whether it’s the right time for an impact evaluation.
• Your decision to use in-house capacity or hire out to conduct an evaluation should hinge on how the costs and benefits balance out in your agency’s context.
• To conduct an impact evaluation, your organization will need to have the following things in place:
  • leadership buy-in,
  • a continuous quality improvement (CQI) system,
  • participant and stakeholder engagement,
  • data and analytic capacity,
  • supportive administrative roles and structures, and

Recommendations

• Agency leadership should value research and evidence generally and your evaluation specifically.
• The intervention should be ready for evaluation and be well established before you consider an impact evaluation.
• The evaluation should be a team effort. The team includes either an internal or external evaluator to design and execute the evaluation but also agency staff, leadership, community stakeholders, and members of the target population.
• Involve stakeholders. Consider the full range of who has a stake in the program you want to evaluate and how best to engage them.
• Do not underestimate the time and resources needed to conduct an evaluation. Agency staff should have significant CQI roles as part of implementing the evaluation. Sufficient funding is necessary to complete your evaluation well.
• Take stock of what you have and consider what you need. Impact evaluations require robust data collection and analytic capacity.

This resource is part of the Roadmaps to Building Evidence in Child Welfare series—a collection of instructional resources about conducting child welfare evaluations. You can find more practical guidance on data, evaluation, and evidence in the child welfare field from CWEST in materials from the Child Welfare Evidence Building Academy, a program of trainings for child welfare agency staff, practitioners, and evaluators.