Artificial intelligence (AI) is increasingly a go-to source for trusted information, but Urban has found that general-purpose AI tools often struggle to accurately answer questions grounded in our data. We are developing custom AI tools built on an AI-ready knowledge base to find out whether an approach that’s specifically designed for models to find, retrieve, and interpret source documents can deliver more-reliable answers.

We’re piloting this work for Urban’s Upward Mobility Initiative (UMI), which produces data and tools and conducts technical assistance to help local leaders apply that evidence to improve upward mobility in their communities. So far, we’ve worked with more than 120 communities, but reliable AI tools could help us support an even wider range of leaders.

We built and evaluated a knowledge base of key upward mobility source documents to test how well AI could deliver accurate, evidence-based guidance. Then, we tested our tool against dozens of realistic questions about upward mobility, spanning topics and levels of complexity, by pointing large language models (LLMs)—Claude Haiku 4.5 and Claude Sonnet 4.6—toward our knowledge base to answer them.

We found that, unlike in previous work, our knowledge base reliably pulled the right source documents and generated accurate responses overall. This success depended on carefully collecting, organizing, and preprocessing source documents so AI tools could better find, retrieve, and interpret them. Our findings indicate that collaboration with experts in the underlying evidence is key for building trustworthy AI tools.

Key elements to creating an AI-ready knowledge base

1. Upfront investment in AI data readiness is crucial.

Our team of data scientists and upward mobility experts curated a set of about 280 documents best suited for answering a subset of expected questions about our upward mobility work. We further invested in the readiness “dirty work”—converting PDFs to machine-readable formats, splitting long documents into component parts, and tagging each document with key information that made it easier to interpret by an AI tool. All these steps enabled our retrieval augmented generation system to much more reliably integrate these sources into responses.

2. Deep subject-matter and local expertise, not AI-specific advances, are key.

To answer realistic user questions (e.g., “What are all of the upward mobility predictors?” or “What strategies can I use to improve upward mobility in my community?”), we did not need state-of-the-art AI. Our UMI team has spent the past decade building resources and providing hands-on engagement. They are the holders of this knowledge. 

The problem we needed to solve was retrieval and navigation. With so many resources, how can an AI tool use only the right ones for a particular question and characterize them the way our experts would want? Alongside the data readiness work, this also meant building a careful set of underlying instructions (i.e., a system prompt), that carefully explained the purpose and proper use of certain documents and prevented the AI models from drawing on outside sources.

3. Evaluation leads to necessary iteration.

Urban’s domain experts created criteria for what constituted correct answers and then used them to evaluate AI-generated responses against those criteria for accuracy, completeness, relevance, and proper use of references. This helped us better understand weak points in our knowledge base, identify the types of questions the models struggled with, and diagnose errors in how certain trusted sources were processed. 

It enabled us to revise and improve our underlying model instructions to be more prescriptive and limit answer wordiness. After two rounds of expert evaluation, we feel confident our tool can reliably help users navigate most questions about the array of trusted evidence our team included in the knowledge base.

What these findings mean for producers of trusted information

These results are promising for organizations like state and local governments, nonprofits, and academic organizations seeking to harness AI’s capabilities while retaining authority on how the trusted information they produce is consumed. We recommend the following guidance for those interested in this approach:

• Data readiness is the most important investment. Upgrading to more expensive models or infrastructure had little to do with our results. Instead, as we continued to refine how UMI resources were encoded and tagged, we saw in real time how much more reliably AI tools retrieved them at the right time and characterized their responses correctly.
• Build with, not for. This phrase is well-known in civic technology spaces and was a core tenet of our work here. Our data science and UMI teams collaborated closely to build this knowledge base with accurate, well-organized sources, design our evaluation suite, and continuously improve the system through expert review and understanding of local leaders’ needs. This work lays the foundation for bolstering, rather than replacing, our existing training and technical assistance.
• Be aware of limitations. Not every question is a good fit for the combination of a knowledge base and AI model. Particularly for nuanced, open-ended questions that require more judgment, our evaluators reported lower scores across the board.
• A well-built knowledge base is key, even in the agentic AI age. Building a reliable knowledge base lays the foundation for our ability to build trustworthy agentic systems. As we look ahead, we anticipate moving toward AI agents (or tools that can act independently on complex tasks) to support stakeholders in using our evidence and resources to support local decisionmaking in highly customizable ways. But any future tools will rely on this knowledge base to be grounded in Urban’s upward mobility resources and experience working with communities.

The barrier to entry to state-of-the-art AI tools has never been lower, at both an individual and organizational level. Yet we find there are no shortcuts to unlocking their true power for changemakers—evidence and human relationships are still fundamental. Our results suggest that with the right collaboration between technical and domain experts, and a commitment to iteration and evaluation, the front door to trusted information can swing open.