THRIVE contracted with the Urban Institute for an external evaluation of the project. Urban developed research questions with the CBO partners. Good research questions are at the heart of high-quality research.

TOOL: The Urban team’s top-level research questions:

1. What are effective methods of transferring cash to households with low incomes?
2. What are the short-term outcomes—as well as the prospective medium- or long-term outcomes—that result from cash supports?
3. How can community-engaged methods be used to build community field research capacity so research can be done alongside communities?

TIP: Tie your research questions to the data sources and indicators you have. Don’t pose questions you can’t answer given the time and resources available.

The evaluation consisted of surveys, qualitative interviews, community-engaged research, and feedback memos.

Surveys

The Urban Institute undertook three surveys that began when the first round of participants received cash and ran through summer 2021.

• Survey 1: All THRIVE respondents had the opportunity to complete this survey, which was focused on the demographics of a respondent’s household, income, and benefits.
• Survey 2: This was offered to anyone who had consented to share their information with the Urban Institute. It included questions on the enrollment process, income, benefits, employment, the economic impact of payments, food insecurity, child well-being, and mental health. To protect the privacy of the respondents, Urban did not tell the CBOs who had completed the survey. Participants received Survey 2 about 90 days after receiving their lump sum or first monthly payment.
• Survey 3: This tool asked about respondents’ experiences and satisfaction with the THRIVE program and the CBOs; their income, benefits, and employment after receiving the full cash transfer; their household’s well-being; and their rent and income stability. THRIVE participants received this survey about six months after receiving their lump sum or first monthly payment.

WARNING: Think hard about the data collection methods you or your external evaluator will use to assess participant outcomes. Consider how these relate to your values and why you need the data. Some questions are needlessly invasive, and participant well-being should be prioritized over data collection.

TOOL: Surveys 1, 2, and 3

Qualitative Interviews

Qualitative interviews add rich context to what survey data tell partners about a project. For the THRIVE evaluation, interviews were conducted with THRIVE staff, government workers who administer social safety net programs, participants, and funders, and they helped the partners better understand participants’ outcomes.

When interviewees mentioned issues they were having, interviewers pointed them in the direction of service navigators. Researchers were prepared with guidance on how to connect interviewees with service navigators.

Community-Engaged Research

The Urban Institute trained five community researchers who live in Ward 8 to conduct the qualitative interviews with participants alongside a staff researcher. Urban worked with FSFSC to recruit locally and train the community researchers, none of whom had previously worked as a researcher. As one Urban researcher noted, the community researchers’ participation “enriches the feedback given about THRIVE, because people being interviewed are more likely to open up to researchers they can identify with and are more likely to be understood by researchers who share a similar neighborhood context.”

The community researchers were paid and received training on the informed consent process and interview protocols. They also practiced interviews and learned where to refer participants who were in distress or facing challenges. After training, each community researcher was paired with an Urban Institute researcher for interviews and on-the-job training and feedback.

TAKEAWAY: The THRIVE evaluation got better data because community researchers conducted qualitative interviews. Participants were talking to people they could relate to.

TOOL: Protocols for community researchers doing qualitative interviews

WARNING: Having neighbors interview neighbors can raise consent and confidentiality issues. During the THRIVE evaluation, the Urban team took steps to make sure interviewers did not know the people they were interviewing.

Community researchers also reviewed the THRIVE interview protocols to ensure they were understandable and sensitive to the community. Community researchers completed their work on the project by contributing their insights to data analysis for the final evaluation report on THRIVE. Urban was setting up its institutional community advisory board, which is made up of community residents who can advise on study designs, while the THRIVE evaluation was under way, and a community researcher was enlisted to advise on that larger process.

TIP: A community advisory board can help researchers understand whether research questions and methods are sensitive to a community.

TOOL: This blog post by Urban Institute researcher Sonia Torres Rodriguez, called “Fostering Resident-Led Research across All Stages of Research,” offers insight on how researchers can better collaborate with residents.

TAKEAWAY: Building a community research team takes a significant amount of time, as does ensuring you are systematically and comprehensively involving the community in research methodology reviews. The preparation is well worth it.

Feedback Memos

The Urban Institute’s evaluators regularly created memos focused on providing real-time feedback to help the CBOs improve their processes. The CBOs discussed the memos at monthly meetings of various partners and weekly operations meetings of implementers. These memos focused on early implementation progress, participant demographics, systems, and fundraising.

TOOL: Funding Direct Cash Initiatives—originally produced as an internal feedback memo for THRIVE partners, this Urban brief explores the following questions: What motivated funders to give to THRIVE? How did THRIVE organize its fundraising?