The voices of Urban Institute's researchers and staff
November 1, 2017

The essential features of a strong federal evaluation office

November 1, 2017

Keith Fudge contributed to this post

The bipartisan Evidence-based Policymaking Commission was authorized by Congress in 2016 to review the current status of evidence building in the federal government and make recommendations for improving the use of strong evidence in policy decisionmaking. The commission’s recent final report proposed ways to improve access to data, protect privacy and confidentiality, and strengthen evidence capacity building in federal agencies.

Yesterday, Speaker Ryan and Senator Murray introduced House and Senate versions of the Evidence-Based Policymaking Act. Its provisions include a mandate that “the head of each agency shall appoint or designate an employee of the agency as the Chief Evaluation Officer of the agency.

This leader would oversee activities to develop and apply evidence, such as using learning agendas to set priorities for evaluations and research with existing resources, improving government-wide coordination of evidence-building activities, and aligning administrative functions and data used in evaluations.

As a former chief evaluation officer at DOL, I’m very familiar with the core characteristics of a mature federal evaluation office. Agencies should consider these details when determining how to build or strengthen a dedicated evaluation unit.

An evaluation officer could be located at the departmental level, like at DOL, or at a subcabinet level, similar to the Administration for Children and Families at the US Department of Health and Human Services. Regardless of the organizational location, critical features should include the following:

An evaluation policy statement. A formal evaluation policy statement must be in effect for the agency or department, emphasizing independence, rigor, relevance, transparency, and ethical practices that reinforce the commitment to evaluation.

Independence. The evaluation office must operate objectively and avoid any appearance of bias or politicization. This requires a clear separation between evaluation activities and responsibility for program operations and policy development.

Decisions about which evaluations to conduct should be aligned with policy priorities and inform program and policy decisions and options, based on scientifically sound, objective evidence. They should not be based on ideological positions or subject to undue political influence.

Procedures for integrity and quality control. Strong, clear procedures must be in place to maintain the integrity of evaluations, including

  • privacy and human subjects protections,

  • development of the evaluation agenda and priorities,

  • technical peer reviews of all aspects of evaluation designs (e.g., methodological design, data collection instruments and procedures, statistical and analytic plans),

  • efforts to minimize the burden of data collection, and

  • release of final reports.

Guidelines for methodological standards. Guidelines that specify standards for high-quality evaluations must be established and followed. These guidelines should be available as a technical resource for both in-house and external evaluators. Likewise, any publicly available evidence-based review system, like a research clearinghouse, must maintain rigorous and consistent quality standards.

Transparent dissemination procedures. Evaluation results should be used to build evidence about what works, what needs improving, and what should be eliminated. Transparent procedures regarding release and dissemination of evaluation reports are essential. This means that all completed reports are released and registries are maintained to keep these reports publicly accessible.

Professionally trained technical evaluation staff. Professional staff with advanced academic degrees and technical skills in economics, public policy, sociology, or related disciplines are essential to a functional evaluation unit. These staff are responsible for many activities, including appropriately designing evaluations, overseeing evaluation projects, and conducting technical reviews. They must also know about and be sensitive to the missions and statutory requirements of programs being evaluated to make sure the studies are relevant.

Although federal evaluation offices and chief evaluation officer roles will look different depending on departments’ needs and capacity, having these core features can help agencies build and use evidence to improve public programs.

Photo via Shutterstock.

SHARE THIS PAGE

As an organization, the Urban Institute does not take positions on issues. Experts are independent and empowered to share their evidence-based views and recommendations shaped by research.