About this episode

We explore how evidence shapes policy, philanthropy, and learning agendas to help drive social change. We also unpack the challenges changemaking institutions can face when incorporating evidence and evaluation into their processes. And we offer insights into how governments and other institutions can foster more trust with the people and communities they serve.
 

Interviewer

Kimberlyn Leary, Executive Vice President, Urban Institute

Guest

Justin Milner, Executive Vice President of Evidence and Evaluation, Arnold Ventures

Transcript

Sarah Rosen Wartell, cohost:
Welcome to Evidence in Action, a podcast from the Urban Institute. I’m your cohost, Sarah Rosen Wartell. I have the honor of being Urban’s president.

Kimberlyn Leary, cohost:
And I’m your cohost, Kimberlyn Leary. I’m executive vice president of the Urban Institute.

Sarah Rosen Wartell:
In this podcast, Kim and I are going to explore the role of evidence: what it is, who makes it, who can use it, who should be using it, and how it can help us to shape policy and achieve better social, economic, and environmental outcomes.

Kimberlyn Leary:
And on every episode, we’ll be joined by a brilliant guest ranging from federal policymakers, local leaders, philanthropists, social entrepreneurs, and those who meet community needs.

Sarah Rosen Wartell:
We’ll be asking them how they use facts, data, and evidence to improve lives and strengthen communities, and also about the limits of these tools in today’s complicated world.

Kimberlyn Leary:
On today’s show, we’ll talk to Justin Milner. Justin is the executive vice president at Arnold Ventures. He has over 20 years of experience focused on the intersection of research, policy, and impact in the nonprofit, philanthropic, and public sectors. And before Arnold Ventures, he was the vice president of the Research to Action Lab right here at the Urban Institute and launched this very podcast.

Kimberlyn Leary:
Welcome to the show, Justin, or I should say, welcome back to the show.

Justin Milner, guest:
Thanks, Kim. I am absolutely thrilled to be here. It’s funny to be on the other side of the microphone.

Kimberlyn Leary:
I imagine.

Justin Milner:
But I’m really excited to join this conversation today.

Kimberlyn Leary:
Well, we’re thrilled to have you. And let me first ask you about your new current role. You’re the executive vice president of evidence and evaluation. Tell us more about that role. How do you direct evidence, and what’s the relationship between evidence and evaluation?

Justin Milner:
So I have been now at Arnold Ventures for eight-plus months and really excited to take on this role and see it very much as an extension of the work that I was doing at Urban Institute. So you said that I was leading the Research to Action Lab, and now I’m taking a step back and focusing more on that research side of things, even as we’re still thinking about how do we really take the research and evidence building that we’re doing through our philanthropic strategies and make it impactful and make it actionable in the real world.

Kimberlyn Leary:
That sounds super exciting, Justin. But when you were growing up, was being the EVP of evidence and evaluation what you were aiming for? What were you interested in when you were younger? What did you focus on and study in college?

Justin Milner:
Great question. I think if you had talked to younger me, college-age me, you would’ve found someone who was deeply invested in thinking about how to impact the world in a positive way. In junior high school, I started an organization, we called it the Colorado Alliance of Environmentally Aware Kids. This was an effort to try to raise awareness about environmental issues. In college, I also had a fellowship my sophomore year to teach at a daycare center for homeless families, and I worked with preschool-age kids there. And so, again, was seeking out opportunities to create positive change in the world through different venues. I think what I didn’t fully realize and didn’t appreciate until later parts of my career is just how hard it is to create positive impact in the world. And that came to this realization that good intentions are not enough to try to truly change trajectories of inequity that existed in our societies.

Kimberlyn Leary:
Well, it sounds like though you didn’t know quite what the roadmap would be. In fact, that motivation and that desire to make a difference led you to have key roles in the philanthropic sector, in government agencies, and at the Urban Institute. So Justin, I hear you saying that when you have motivation and when you have scale, you can have uncommon momentum. Would that be a fair way to describe it?

Justin Milner:
I like how you’re thinking in terms of this algorithm. I think, let me add to it.

Kimberlyn Leary:
Please.

Justin Milner:
I think motivation is critical. I think scale affords the possibility for tremendous learning and tremendous impact. And I would add to the algorithm, rigor. And I think the importance of bringing a rigorous approach to how we’re learning from what we’re doing is essential such that we can draw the right sorts of conclusions, that we can learn as well as possible the kinds of lessons that can really drive better work and more improved service provision and program and policy in the future.

Kimberlyn Leary:
I appreciate that more precise friendly amendment. So Justin, what do people in general not know about the way in which government agencies work in terms of their research and evidence agendas?

Justin Milner:
I would hazard to guess most people know very little about what government agencies are doing to learn from the programs and policies that they support. I think people might assume that these types of learnings and research is happening in the background but might not have a great sense of what that looks like. I think what people might be surprised to learn, particularly in the wake of the Evidence Act of 2019, is how intentional most federal agencies are in thinking about developing learning agendas and thinking about developing evaluation portfolios and thinking about how to tie some of the work that they’re doing in building a knowledge base about what is effective and using that to inform policymaking. I think they might also be surprised at just how difficult and diffuse that process can be in practice.

Kimberlyn Leary:
And all of those components that you mentioned, learning agendas, of portfolios, that’s all part of the Evidence Act, right?

Justin Milner:
That’s all part of the Evidence Act and those are things that were jump-started through that law. And what the Evidence Act has done has given a more formal channel and a formal directive to agencies across government in places that were doing that work and in places that weren’t, to get on the evidence train and start to build what those learning agendas could look like and think more strategically about how it can inform their work.

Kimberlyn Leary:
So let me carry that further just a little bit. In your opinion, what are the biggest challenges government agencies face in incorporating the evidence that they’re now gathering and assessing into the policymaking process, and how can those challenges be addressed?

Justin Milner:
So this is a phenomenal question and one that I don’t think has easy answers, but let me point out three things that I think are relevant. One is demand. We can create a supply of evidence and invest in research, but there needs to be a demand for that research and a place for it to go to really inform policymaking. And I think it’s on us, and I mean that broadly, Urban Institute, Arnold Ventures, to continue to support the development of the ecosystem that incentivizes policymakers and decisionmakers actually relying on and using the data and evidence that are coming out of their agencies. I don’t think that’s a given and it’s a hard nut to crack.

I think a second thing are timelines. The reality is that for many programs and evaluations of such programs, the duration that you want to study a program and study its impacts, you don’t want to know: Did this work? Did this job training program work after three months? You want to know: Did people stay in a job? Did it affect their earnings? Did it do that multiple years out? So the timelines can be really challenging to align with, what is the question that we’re asking now and what’s the timeline for when we’re going to be able to provide the evidence? That’s a real challenge.

And then a third is resources. The reality is, is that we pour out billions and billions and billions of dollars to support programs and policies for which we have absolutely no sense of whether they are effective or not. And so we really should be thinking about a legislative requirement, for example, that for particular departments, they are spending 1 percent of program dollars, 1.5 percent, even 0.5 percent of program dollars on evaluation to support learning and research.

Kimberlyn Leary:
Let me ask you though about, some not so much “best practices,” that’s policy speak, but where have you seen initiatives or perhaps some strategies that you’ve observed during your career where there has been successful bridge building between research, policy, and practice in the public sector?

Justin Milner:
I think one of the things that I would speak to that I’ve both worked on and seen from afar as being important in catalyzing momentum has been the development of cohorts, and having cohorts of leaders who sit in a similar place and maybe facing similar challenges. So one cohort that comes to mind is What Works Cities. And so this has been a collaborative that has been supported by Bloomberg Philanthropies and Results for America, the Behavioral Insights Team, and others. And they have worked to identify specific challenges across those different cities, it’s like 60-plus cities, and bring the research to bear on those specific challenges and have also created shared learning environments where leaders, mayors often, are able to talk with each other and learn from each other’s approaches and strategies that they’ve taken in their own context.

Another that I’d highlight is work from the Urban Institute that we helped to lead. We have done work thinking about, what does upward mobility look like and how do we support upward mobility at a high level in communities across the country? One of the things that Urban has done that I think has been incredibly powerful was to develop and detail a set of indicators, 35 indicators, that communities and cities and localities could track to get a sense of how much and how well those communities are supporting upward mobility for their citizens. And so having that shared sense of data, the shared sense of indicators, the shared sense of questions that one might be asking is a way to advance the work in those places, and again, identify what are some of the concrete things that are going to be specific to their needs.

Kimberlyn Leary:
I also think that the role of collaboration is so critical in the cohort models that you’ve mentioned, because not only can cities learn from one another and not only can mayors share practices, they may also raise the next generation of questions for which new research is needed.

Kimberlyn Leary:
How does your experience at Arnold Ventures differ from your time in the public sector in terms of developing evidence, and what unique opportunities does philanthropy have to advance evidence-based policy?

Justin Milner:
So I think it differs in a couple of ways. One is that there is much greater latitude working at a private philanthropy than there is within a government context.

Kimberlyn Leary:
Latitude in terms of?

Justin Milner:
In terms of, I often thought of my time in government, in which I was earlier in my career, as being in a layer cake. And there were many layers to that layer cake, many layers up, maybe a couple of layers down. I was early in my career again, so there weren’t that many layers down. But you are set within that—I’m going to torture the metaphor now—but set within that cake and there’s a lot of heft to it. And for things to truly reach the surface, to get to the top, would have to go through a lot of... Okay, I’m really losing the metaphor there. Maybe I should think of a different metaphor.

Kimberlyn Leary:
Go through the chocolate and the vanilla and the strawberry to get to the frosting.

Justin Milner:
Yes. The larger point is, there’s a lot of actors involved, there’s a lot of vested interests, and there’s a lot of diffuse decisionmaking authority. And within philanthropy, that’s much less the case. So there is much more latitude to identify opportunities, try to be nimble to meet those opportunities and try to support things as quickly as possible. An example of that is work that Arnold Ventures has done in gun violence research. And for many years, through the federal government, really until very recently, we supported little to no research on gun violence. There are questions here that are important research questions, important policy questions and important questions for our country to grapple with about the impacts of guns, gun violence. And we supported the development of an organization to start diving into that white space and has done tremendous work. That’s a unique place that philanthropy can step into.

Kimberlyn Leary:
Justin, you’ve been affiliated with a number of distinguished institutions, starting from your alma maters of Yale and Princeton to the Urban Institute and now Arnold Ventures. But right now, many people in communities in this country and globally don’t trust institutions as they once did. How do you understand that? What has taken place that puts a cloud of suspicion or distrust in play? And what are a couple of things that institutions could do to demonstrate that they are indeed trustworthy?

Justin Milner:
So this is a huge question, and it’s one that is almost foundational to our society at this particular moment, a place in which people have radically different versions of an America that they experience. And I think part of this distrust of institutions comes out of an increasing inequity that people have been experiencing for the last several decades in this country. The reality that it is increasingly hard to join and stay in the middle class and feel like you have the kinds of opportunities that you saw your parents have. We have seen increasing aggregation of wealth. We have seen opportunities available to some but not to most. And I think people experience that on a day-to-day, visceral basis. When it feels like institutions may not be helping to level the playing field and may not be helping to change people’s experiences, again, some of this may be fully based in reality and some of it may be experiential, but if people don’t see institutions really trying to change that and level the playing field for people from different backgrounds, I think you start to distrust what are the purpose and goals of those institutions.

I think from a research and evidence-based perspective, which can seem very wonky and can seem like it’s not even a part of the conversation a lot of the time, I think there are some core tenets to this work and to this frame that can be valuable in that type of level setting. One is for government and institutions to be more transparent, to not only potentially collect data, but to share out that data in a way that can be meaningful for people, not in a way that people feel like their privacy is being taken or that they are being commoditized on a daily basis, but sharing back such that we have a better sense of where things are going well and where things are not going well. So I think that transparency hasn’t been there.

I think also institutions could do a better job of trying to demonstrate what it is we’re trying to achieve and what we’re trying to learn to improve lives. And those are difficult messages, but it’s not something that we anchor on and that larger sort of shared vision sometimes. And I think that’s another place in which the work of institutions and government can break down.

Kimberlyn Leary:
I know that some of the work that you did when you were at the Urban Institute and the Research to Action Lab also was aimed at engaging with communities. I know their models and community-engaged methods and community science, where exactly what you mentioned, is another tool and technique to make it possible for communities to partner with institutions, to recognize that they have shared goals and to recognize that different kinds of expertise can be found in communities, that it can be quite important to understanding the evidence that’s produced through research and through data analysis.

Justin Milner:
It’s a fabulous point. And I think about it often in multiple stages, that there can be that front-end engagement in which you are working with communities, talking with people about the research process and the research questions. That they can, they themselves, even what we’re asking and looking to learn, should be informed by people’s experiences there. And then you’re thinking about it at the back end and thinking about how are we sharing back what we’ve learned, how are we doing so in ways that’s going to be compelling, communicable, and relevant for people’s lives.

Kimberlyn Leary:
And government has started to recognize this in many areas by focusing on customer experiencing. There was an executive order a few years back about that. And also with the equity executive orders, a real attention to trying to engage with communities and stakeholders. So things are moving in that direction, yeah.

Justin Milner:
And I love that, the customer experience work, which, I think, it’s a phrase that feels unfamiliar in a government context, but it’s important to bring some of those principles that we know from other places to bear. And it hits upon this nexus of programs and policy and what we want to do and the implementation and actually getting it done. And that part of it I think has been long underappreciated, but is starting to come to the fore with examples like that.

Kimberlyn Leary:
So we like to ask each visitor to the show the very same question, and here it is. Do you have a favorite example where high-quality evidence has informed choices that result in making the lives of people better? Can you share with us an example of evidence in action?

Justin Milner:
Evidence in action, I love this! I think this is the title of the podcast.

Kimberlyn Leary:
Yes, it is.

Justin Milner:
So there’s a great example that we’re really excited about at Arnold Ventures, and that’s called ASAP, Accelerated Study in Associate Programs. This is a program that was started in CUNY, City University of New York, and has since been replicated in Ohio. We have supported other replications in other states and is about to go really big in New York state. This is a program that supports students that are in community colleges, two-year programs, to help them when they start the program to actually finish the program. What this intervention has found has been able to raise the graduation rates significantly by 11 to 15 percentage points, which is even a much higher percentage increase of graduation, and has been able to increase long-term earnings significantly.

So six years later, students that were in this program not only finished their community college degrees at a much higher rate, but they’re also earning a lot more money. They also recently tested and proved impact for this program for students in four-year programs in New York. This is the kind of program that could apply across the country in all states and really thinking about different community college contexts as a true good, getting students to the finish line, and making sure they end up in jobs that pay more. That is evidence in action and one of the places that AV is really thinking about how we might scale that impact in the future.

Kimberlyn Leary:
So exciting, Justin. And yes indeed, that’s the kind of impact I think that we’re all aiming to have. So the last time that we were on camera and wearing headphones and had podcast mics in front of us, you were interviewing me. I hope I’ve done you proud as the moderator now of this podcast, but we are just so thrilled that you could join us, and we want to thank you for being on the show.

Justin Milner:
Kim, thanks so much. It has been an absolute pleasure getting to talk with you even on this side of the microphone. And I’m absolutely thrilled to know that Urban is relaunching the podcast.

Kimberlyn Leary:
Join us next time on Evidence in Action as we have conversations about important ways that drive change with our talented and captivating guests. And if you would like to learn more about us, go to our website at urban.org. You can also follow the show on Apple Podcasts, Spotify, Amazon Music, and wherever you listen to your podcasts. This has been Evidence in Action, created by the Urban Institute and Pod People. I’m your cohost, Kim Leary. Thank you.