AI development tools have proliferated, making it easier than ever to spin up a working AI solution in just a few hours.

But there’s a big difference between a tool that works and a tool that works well.

Building reliable, high-quality AI tools requires investing in AI-ready formats for data and working with subject area experts who understand what specific guidance an AI model will need to properly use that data to answer users’ questions. In other words, both technical and domain expertise are critical to fine-tuning AI tools to match their given purpose. And that’s exactly what the Urban Institute AI Studio brings to the table.

Here, we outline three lessons we learned while building an AI navigator tool for the Urban Institute’s vast library of upward mobility data, tools, and technical assistance resources. By bringing together data scientists and upward mobility experts, we created a tool that provides users accessible, actionable answers to their upward mobility policy questions, grounded in Urban’s robust evidence base.

1. Subject matter experts are critical to curating an AI tool’s knowledge base.

To produce accurate, reliable results, an AI tool does not need every resource on a given topic. It needs access to the right resources.

Rather than dumping all of Urban’s upward mobility resources into the tool’s knowledge base, our team of subject matter experts curated a set of questions users were likely to ask. These questions ranged from simple (“What are all of the upward mobility predictors?”) to complex (“What are promising local interventions for jobs paying a living wage, and what have the effects been of these initiatives?”).

We then had dozens of researchers select the Urban resources (280 total) best suited to answering the questions. Their expertise helped ensure the AI tool would provide answers grounded in upward mobility planning best practices, promising local interventions, and data-driven decisionmaking.

The AI Studio can help mission-driven organizations set your AI tools up for success by mapping realistic use cases and curating the essential information the tool needs.

2. Investing in AI-ready data can improve performance.

Once we identified what resources would form the AI navigator tool’s knowledge base, we had to make them AI ready.

To make longer documents easier for the AI to parse, we programmatically split them by chapters or sections. Doing so manually wouldn’t have been feasible or efficient at scale, and we had discovered during a previous project that automatically splitting a long document improved retrieval.

We have that thoughtful metadata tagging significantly improves how AI tools retrieve and use information. Where possible, we automatically extracted and applied tags for information like the year of publication or the type of information in a document (i.e., data versus a case study).

However, early testing found the model would benefit from adding tags that provided the AI with context on how to interpret each document. This context-specific tagging required the expertise of our research colleagues and a manual tagging process. For example, we created a metadata tag that indicated whether a document contained evidence, summary information, or local interventions.

We also found that building deep linking into our Upward Mobility Initiative Dashboard made it easier for AI tools to cite it. We designed the dashboard so every view could be mapped to a unique URL, which made it easier for us to write citation instructions for the navigator. Our policy and technical experts can work with your staff to design your data systems and tools to be AI ready from the start. We can also help preprocess existing information to create a knowledge base that is fine-tuned to your AI solution use case.

3. System prompts and skills can help guide AI tools to quality responses.

Though the AI navigator tool’s ability to retrieve the correct information steadily improved with each step of preprocessing, we found it sometimes removed relevant documents from consideration. That’s because initially, we were using intelligent metadata filtering, where a separate large language model (LLM) filters the universe of relevant documents based on metadata tags. If a user asked a question about “minimum wage,” it would surface only exact matches in the knowledge base.

To solve this issue, we gave the navigator a detailed system prompt that instructed the model on how to retrieve and analyze documents of different types based on the nature of a user’s question. This underlying set of instructions would be appended to every user question to guide the navigator toward our desired tone, style, and structure for the tool; requirements around careful citation; and exemplary responses.

With the system prompt in place, if a user asked a planning or process question about how their community can boost upward mobility from poverty, the model would prioritize documents tagged with the “toolkit” document type, connecting the user to Urban’s Toolkit for Increasing Upward Mobility in Your Community.

Building on the navigator pilot, the Urban team is developing an agentic AI tool that can summarize Urban’s upward mobility data and evidence into actionable products customized to the user’s interests and community. Our technical and subject matter experts are creating agent skills that capture our best guidance on how the agent can analyze and interpret the data and produce accurate and useful results.

At the AI Studio, our experts work with you to create guidance, such as system prompts and skills, that are specific to your use case and that reliably guide an AI tool toward your desired outcomes.