In 1997, Geoffrey Canada set about tackling poverty in Harlem by focusing on the neighborhood’s children. Today, his 97-block Harlem Children’s Zone is a cradle-to-diploma network of schooling and other supportive programming designed to help kids from families with low incomes succeed in college and beyond.

Impressed by this model, in 2010, the U.S. Department of Education began offering grants to encourage other communities to create children’s zones, or “Promise Neighborhoods,” of their own. But can an idea born in upper Manhattan work in western Kentucky? What counts as success for such an ambitious undertaking, and how do you measure it? The Department of Education asked the Urban Institute to answer that last question. “Measuring Performance: A Guidance Document for Promise Neighborhoods on Collecting Data and Reporting Results” was released earlier this year. I sat down with “Measuring Performance” co-author Peter Tatian to talk about the challenge of quantifying the impact of a long-term, multifaceted program.

Promise Neighborhoods provide resources for children and parents over many years. There are literally hundreds of different outcomes grantees could measure for thousands of people. How did you decide where to begin?

Part of what an organization agrees to when it accepts Promise Neighborhoods funding is that it will report on 15 Government Performance Results Act (GPRA) measures. For example, one of the measures asks, does a child have a medical home, that is, a doctor or some other certified healthcare provider, whom he or she can see on a regular basis? Other indicators focus on early childhood development and academic standards. These 15 GPRAs were our starting point.

What were some of your primary considerations as you tried to translate these measures into a robust data program?

First, the grantees are diverse. Some are community-based non-profits. Others are attached to universities. When it comes to data gathering, they have different strengths and face different challenges, so we wanted to leverage information sources that would be accessible to all of them.

Secondly, we wanted to identify data that they can collect repeatedly, because they have to measure progress over time.

And where possible, we used existing sources, like standardized test scores.

Some of this is completely new to them.  Because collecting and using data is, essentially, research, Promise Neighborhoods have to do things like have their research plans approved by an institutional review board (IRB) [a body that screens research involving human beings for potential ethical issues]. The IRB process is second nature to researchers, but for many community-based groups, it’s not necessarily something they’ve done before.

This is not easy work. People have been trying to solve these issues for generations, and I think our partners understand, if they don’t do a good job tracking the results, this could just be another well-intentioned effort that doesn’t lead anywhere.

You’ve studied or provided technical assistance for other federally funded community development initiatives. Are the reporting requirements for Promise Neighborhood grantees especially demanding?

I think they are. The folks we’ve worked with at the Department of Education have commented that the 15 GPRA measures written into the Promise grants are the most they’ve seen in any program with which they’re familiar.

But these measurements have proven very useful in helping focus the grantees’ resources and attention. For example, one of the first things the GPRAs ask you to assess is whether three-year-olds are exhibiting “age-appropriate functioning,” that is, timely cognitive development. In some places, the percentage of children who have reached this benchmark is small. A grantee will, naturally, ask, why is that? Is it because too few area three-year-olds have access to early care [e.g. Head Start and other pre-school programs]? Maybe a lot of them are in early care, but the quality is insufficient. How do you fix the problem today and make sure children who will turn three a few years from now will be significantly better off?

A single question spurs action across many areas. In this case, a focus on three-year-olds forces communities to think about how they can best meet a range of early childhood development needs. We have a term for this now: “GPRA discipline,” this process of thinking rigorously about specific outcomes and what it will take to achieve them.

You don’t have to be a Promise Neighborhood to ask these questions in your community.

Not at all.

So is it realistic to think that this type of measurement and needs assessment could spread beyond the funded areas?

That’s the hope. It’s a model many hope will prove to be beneficial and that will be replicated in the form of additional Promise Neighborhoods and in other initiatives, which use some of the things we’ll learn from Promise.

Once the data on Promise Neighborhoods becomes available, who will be interested in it?

I think lots of people will be: the Promise Neighborhoods grantees themselves, the Department of Education. The federal government and other funders need to be able to show this investment is yielding results.

The people who live in these communities are another important consumer of these data. They have the right to know whether the Promise Neighborhood is achieving the kind of results they’d like in their community.

And, finally, there’s the research and policy world more broadly. We know a lot about how to evaluate individual programs, but here, the hope is that the whole is bigger than the sum of its parts—and there are a lot of parts. The frameworks and techniques researchers develop to tease out the impact of individual elements of Promise Neighborhoods could shape the way we examine place-based initiatives for years to come.

Students image from Shutterstock