Beyond Chatbots: How to Build AI That Understands Your Organization’s Knowledge

There is a big difference between giving artificial intelligence access to your content and building an AI system that actually understands how that content fits together.

We recently had an organization approach us with an interesting challenge.

They had a specific curriculum they wanted integrated into an AI-powered system. The goal wasn’t simply to upload a collection of documents and create a chatbot that could answer questions about them. They wanted the system to understand the curriculum well enough to make meaningful connections: between topics, concepts, resources, standards, activities, and other instructional materials.

In other words, they didn’t just need AI that could find words. They needed AI that could reason over a structured body of knowledge.

That distinction led us down an interesting technical path—one with implications far beyond education.

The Problem With “Just Give It to the AI”

Imagine that your organization has spent years developing specialized knowledge.

Maybe it is a curriculum. Maybe it is a collection of evidence-based interventions, clinical guidance, organizational policies, implementation resources, technical assistance materials, or hundreds of research reports.

You would like someone to be able to ask: “I’m working on this problem. What in our knowledge base is relevant, and how should I use it?”

Modern large language models are remarkably good at producing plausible answers to questions like this.

But plausible is not necessarily the same as correct.

That becomes especially important when the source material represents an organization’s carefully developed intellectual property, scientific evidence, professional standards, or established guidance.

A generic AI system might know something about your subject.

It does not necessarily know your organization’s understanding of that subject.

That was the challenge we needed to solve.

From a Library of Content to a Map of Knowledge

Our first step was not to build a chatbot. It was to build a map.

We took the organization’s curriculum and represented important pieces of information as a knowledge graph.

A knowledge graph sounds complicated, but the basic idea is simple.

Suppose your organization knows that:

Topic A → relates to Concept B → which is supported by Resource C → which can be taught using Activity D.

A traditional database might store those four things.

A knowledge graph also stores the relationships between them. That matters because humans rarely think about knowledge as isolated documents. We think through connections.

A public health professional might think: I’m working on overdose prevention. What interventions are relevant? Which populations have they been tested with? What implementation barriers should I anticipate? What measures could I use? What resources would help my community implement this?

Each question moves through a network of related knowledge. We wanted the AI system to be able to do the same thing.

Teaching a Computer What “Related” Actually Means

This required us to define different kinds of relationships within the organization’s content.

Not every connection means the same thing.

One resource might support a concept.

Another might teach it.

One topic might require knowledge of another.

A resource might align with a standard.

An activity might extend something already being taught.

This turns a pile of content into something much richer: a structured model of how the organization understands its own domain.

And that opens up an entirely different kind of AI application.

Instead of asking a language model to invent connections from scratch, we can first ask the knowledge system:

What does the organization’s own evidence say belongs here?

Then the AI can help explain those connections in useful language.

Search Isn’t Enough Either

There was another problem.

People don’t necessarily ask questions using the terminology contained in a database.

Someone might type: “I’m teaching this next month. What could I connect it to?”

The relevant resource might never contain the exact words in that question.

So we combined the knowledge graph with semantic search.

Semantic search looks for similarity in meaning, not simply matching words.

Think about the difference between searching a filing cabinet and asking a knowledgeable colleague.

A filing cabinet requires you to know where something was filed.

A knowledgeable colleague can say: “I know you didn’t use this exact terminology, but I think this is what you’re looking for.”

Semantic search gives the system some of that flexibility.

But semantic search alone has weaknesses too. Something can sound similar without actually being the right connection.

That’s why we combined both approaches.

Semantic Search + Knowledge Graphs

This became one of the most important pieces of the project.

When someone asks a question, the system can first identify content that is semantically close to what they mean.

Then it can explore the organization’s defined relationships around that content.

In simplified form: Question → Find conceptually similar material → Follow meaningful relationships → Rank the strongest connections → Build an evidence set → Generate an answer

This is known as hybrid retrieval.

The semantic component provides flexibility. The graph provides structure. Together, they help overcome weaknesses that either approach would have on its own.

Not Every Question Is Looking for the Same Thing

We also realized that relevance depends on what the person is trying to accomplish. Consider these questions:

• “What should I teach about this?”
• “What activity could I use?”
• “What standard does this address?”
• “What should I read before teaching this?”

They might all concern exactly the same topic.

But the best answer is different because the intent is different.

So the system first classifies what kind of question the person is asking.

It can then change how it searches and ranks the knowledge base.

This is an important lesson for organizations considering AI applications.

A useful AI system shouldn’t merely ask: “What content resembles this question?”

It should also ask: “What is this person actually trying to accomplish?”

That brings us much closer to useful decision support.

Then We Tried to Break It

We didn’t want to build the system, try five questions, declare victory, and move on.

We created a benchmark.

For each test question, we specified what information the system should retrieve.

Then we measured whether it actually did.

We tested hundreds of queries across different topics, months, and types of questions.

Our final reference retrieval system achieved:

• 92.3% average recall within the first five results
• 97.5% average recall within the first ten results
• 95.8% of test questions retrieving all expected material within the first ten results

Those numbers aren’t measures of how eloquent the AI sounds.

They measure whether the underlying system is retrieving the knowledge it is supposed to retrieve.

That distinction matters enormously.

Why We Froze the System

Once the retrieval system performed well enough, we did something that might sound strange.

We stopped improving it. More precisely, we froze that version.

Why? Because continually tuning a system against the same test questions eventually makes the test less meaningful. You can inadvertently optimize for your benchmark rather than for the real world.

So the validated version became a reference point.

We then created a separate set of tests for moving the system from our research environment into a production web application.

For ten representative queries, we recorded not just the expected answer, but the underlying retrieval behavior: which items appeared, in what order, and with what scores.

The production implementation now has to reproduce that behavior.

There are 45 automated checks associated with that translation.

The standard isn’t: “It seems about right.”

It is: “Does the production system behave like the system we actually validated?”

That’s a very different approach to AI development.

The AI Is Not the Knowledge Base

Perhaps the most important design decision is also one of the simplest.

We don’t want the language model to decide what the organization believes.

The organization’s content does that.

Our working principle became: The AI can be creative about how it communicates and applies the knowledge without being creative about what the underlying knowledge says.

That means separating two jobs.

First, the retrieval system determines what evidence is relevant.

Then the language model turns that evidence into something useful for a human.

We can subsequently examine the generated answer and ask:

• Which claims are directly supported?
• Which are reasonable applications of the evidence?
• Did the AI introduce something that wasn’t actually in the retrieved knowledge?

If it did, the system can repair or reject the answer rather than confidently displaying it.

Why This Matters for Public Health

Although this particular project began with a curriculum, the architecture is much more general.

Public health organizations are surrounded by specialized knowledge.

Think about what lives inside a health department, university center, nonprofit, research institute, professional association, or technical assistance center:

guidelines, toolkits, evaluation reports, surveillance data, implementation resources, training materials, evidence reviews, measures, case studies, policies, grant deliverables, FAQs, and years of institutional experience.

The problem usually isn’t that the organization has no information.

The problem is getting the right information to the right person when they need it.

Imagine asking: “We’re a rural health department trying to improve vaccination uptake. What resources do we have that would actually apply to our situation?”

Or:

“A coalition is struggling with implementation. What barriers have we seen before, and what strategies have worked?”

Or:

“We’re writing a grant around youth mental health. What evidence, measures, programs, and previous work in our organization are relevant?”

Those aren’t merely search questions. They require connecting knowledge.

Your Organization May Already Have the Hardest Part

One of the more interesting lessons from this work is that organizations sometimes think they need to “get ready for AI.”

But many have already spent years creating the most valuable ingredient.

They have the knowledge.

It’s sitting in PDFs.

It’s sitting on websites.

It’s in spreadsheets and databases.

It’s embedded in training programs.

It’s spread across technical assistance resources.

It’s represented in the expertise of staff who know that this resource goes with that problem, except under these circumstances.

The opportunity is to make those relationships computationally useful.

That can mean moving from: documents → structured knowledge

from: keyword search → meaning-based retrieval

from: isolated resources → connected evidence

and from: generic AI → organization-specific intelligence.

What We Learned

This project changed the question we ask when thinking about organizational AI.

The question isn’t: “How can we put a chatbot on this website?”

That’s relatively easy.

A much more interesting question is:

“How can we build a system that understands this organization’s knowledge well enough to help people use it?”

That requires more than a language model.

It requires understanding the content domain, modeling relationships, designing retrieval logic, testing it against known answers, measuring performance, controlling what the AI is allowed to claim, and translating the validated system into software that can actually be maintained.

But the result is also much more powerful.

You’re no longer simply asking AI what it knows.

You’re building an interface to what your organization knows.

And for organizations sitting on years—or decades—of specialized knowledge, that may be one of the most interesting applications of AI yet.