Voice AI Agent Knowledge Base: What Goes in the Prompt, What Goes in Retrieval, and How to Stop Made-Up Answers (2026)

TL;DR: A voice agent can pull an answer from three places: the prompt it was built with, a knowledge base it searches mid-call, or a live tool call to a real system. Most agencies dump everything into one of the three and then wonder why the agent is slow, wrong, or both. My split: behaviour and the fifteen answers callers actually ask go in the prompt, the long tail of documented facts goes in the knowledge base, and anything that changes hour to hour goes in a tool call. Then you write an explicit refusal rule, because an agent that confidently invents a price is worse for your client than one that says it will have someone call back. I build production voice agents for US clients on Retell, n8n, GoHighLevel, and Twilio, and I run VoiceDash, the white-label client portal agencies hand to their clients.

The knowledge base is the part of a build clients think is easy. They send you a 40-page PDF of their service menu, an old FAQ page, and a Google Doc someone wrote in 2023, and they assume the agent now knows their business. What they have actually sent you is three sources that contradict each other, and the agent will pick one at random on a live call.

The three places an answer can come from

Every question a caller asks gets answered from one of three sources, and choosing wrong is what causes most of the quality complaints I see.

The prompt is what the agent always knows. It costs nothing to access, it is instant, and it is what the model reasons over on every turn. It is also the most expensive place to add words, because every token you add there is processed on every single turn and dilutes attention from the rules that control behaviour.

The knowledge base is what the agent looks up when it needs to. It scales to hundreds of pages without slowing down the greeting. But retrieval is a search, and a search can return the wrong chunk, or nothing.

A tool call is what the agent asks a real system in the moment: is that slot open, is this customer in the CRM, what is the balance. It is always current and it is the only one of the three that can be, but it costs you a live round trip mid-call, which is exactly the pattern I warned about in how to reduce voice AI latency.

What belongs in the prompt

Two things: how the agent behaves, and the small set of facts that come up on most calls.

Behaviour is non-negotiable prompt material. Persona, greeting, AI disclosure, escalation triggers, confirmation habits, and refusals all live here. That is a separate craft, and I covered the phrasing that actually works in the voice AI receptionist prompt guide.

The facts worth hardcoding are smaller than clients expect. Pull the client's last 100 calls, or just sit with whoever answers the phone for an afternoon, and you will find that ten to fifteen questions cover the overwhelming majority of them. Hours. Address and parking. Do you take my insurance or my brand of equipment. What is the price range for the one thing everyone calls about. Do you charge for the callout. How soon can someone come.

Those go in the prompt verbatim, because they are asked on most calls and you do not want a retrieval round trip in the middle of the most common exchange in the business.

Everything past that list is a knowledge base problem.

What belongs in the knowledge base

The long tail: the 200 things a caller might ask that are documented somewhere and change a few times a year.

Full service descriptions. Individual procedure or product detail. Warranty terms. Financing options. Which brands are serviced. Preparation instructions before an appointment. Cancellation policy detail. Areas covered outside the main radius. Staff bios if the vertical needs them.

The test I use: if it is written down somewhere, does not change this week, and comes up on fewer than one call in ten, it goes in the knowledge base. That single rule resolves most arguments about where something belongs.

What belongs in a tool call instead

Anything with a current value is not a knowledge base entry, no matter how much it looks like one. Appointment availability, order or job status, whether a specific customer exists, outstanding balances, today's wait time.

Agencies get this wrong in a very specific way: the client says "our availability is Tuesdays and Thursdays," it gets written into a document, and four months later the agent is still telling callers about Tuesdays while the calendar disagrees. Current-state answers come from the calendar or the CRM through a tool call, and the plumbing for that is the same pattern as connecting Retell AI to n8n.

Write documents for a voice, not for a web page

This is the step almost everyone skips, and it is where the biggest quality jump is available.

A knowledge base article written for a website is written to be scanned by an eye. Retrieval hands a chunk of it to a model that has to turn it into something a human hears once, with no ability to re-read. Those are different formats.

What I do to every document before it goes in:

• One topic per document. A single page covering pricing, hours, and insurance will get retrieved for all three and answer none of them cleanly. Split it.
• Lead with the answer. The first sentence should be the direct response, because that is the part most likely to survive into what the agent actually says.
• Kill the layout. Tables, price grids, and nested bullets read terribly out loud. Rewrite them as sentences: "a standard cleaning is $120 and takes about an hour."
• Write the way the caller asks. People do not call and say "elucidate your onboarding procedure." Title the document with the actual question. This matters for retrieval quality, because the caller's phrasing is what gets matched.
• Expand everything. No internal abbreviations, no acronyms, no "see above." Each chunk has to stand alone, since the agent may only get that one piece.
• Spell out anything spoken. Phone numbers, addresses, and dollar amounts should be written the way you want them said.
• Delete stale documents rather than updating around them. Two versions of the cancellation policy in a knowledge base is worse than none.

Aim for short documents, a few hundred words each. Long documents get chunked in ways you did not choose, and you find out mid-call.

Stopping made-up answers

Retrieval reduces hallucination. It does not eliminate it, and voice makes hallucination worse than chat does, because there is no source link next to the answer and nobody is reading critically. The caller just hears a confident sentence and believes it.

Three controls I put in every build:

An explicit refusal rule in the prompt. Something along the lines of: if the answer is not in your instructions or the knowledge base, do not estimate, do not generalise from similar services, and do not reason from what is typical for this industry. Say you want to make sure they get the exact answer, and take a message or transfer. Models are helpful by default, and helpfulness is precisely the failure mode here.

A hard no-improvise list per vertical. Prices not in the documents, timelines, anything that sounds like a guarantee, anything that could be read as medical, legal, or financial advice. In regulated work this is the whole ballgame, which is why voice AI for law firms needs tighter scope refusals than an HVAC overflow line.

A route for every refusal. A refusal that dead ends is a lost customer. Every "I do not have that" needs somewhere to go: a callback commitment written to the CRM, or a live transfer per how to transfer a voice AI call to a human.

Then test it. Deliberately ask the agent five questions you know are not documented and listen to what it does. That belongs in the standing regression set from how to test a voice AI agent, because prompt edits change refusal behaviour in ways nobody predicts.

The maintenance problem nobody plans for

A knowledge base is accurate on launch day and decays from there. The client raises prices, drops a service, adds a location, changes the cancellation window, and tells nobody. Three months later the agent is quoting last year's prices to real customers and the client blames the agent.

What works:

• Put knowledge base updates in the retainer explicitly, with a stated turnaround. It is one of the few tangible deliverables that justifies a monthly fee, and it belongs in the kickoff conversation covered in how to onboard voice AI clients.
• Give the client one channel to request changes. One. Not texts, emails, and passing comments on calls.
• Date-stamp every document with when it was last confirmed, and re-confirm the price-bearing ones quarterly.
• Read transcripts weekly for questions the agent could not answer. That list is your update queue, and it is more reliable than anything the client will think to tell you.

Proving the knowledge base is working

That last point only works if you can actually see the calls, and if the client can too.

This is where the portal earns its place. In VoiceDash, every call the agent handles lands in a branded client portal with your logo on your domain: searchable timestamped transcripts, recordings, AI summaries, live call volume and outcomes, scoped so each client only ever sees their own workspace. When you spot three callers in a week asking about a service that is not documented, that is a knowledge base gap you found before your client did, and you can point at the transcripts when you bill for fixing it. It connects to your Retell account, goes live in under 10 minutes with no code, and plans run $19, $49, and $99 a month with a 7-day free trial and no card required. VAPI and Bland support are coming soon; today it is purpose-built for Retell agents.

Handing the client that view instead of a monthly summary email is the same argument I made in voice AI client reporting: visible work is billable work.

The pre-launch checklist

1. List the top fifteen caller questions from real calls, not from the client's guess, and hardcode those in the prompt.
2. Move everything documented and rarely asked into the knowledge base, one topic per document.
3. Move every current-state answer into a tool call and remove it from the documents.
4. Rewrite each document for the ear: answer first, no tables, caller phrasing in the title.
5. Delete every stale or duplicate source, especially old PDFs.
6. Add the refusal rule and the vertical's no-improvise list to the prompt.
7. Ask five undocumented questions on a real call and confirm the agent refuses and routes.
8. Date-stamp the documents and put the update process in the retainer in writing.

The bottom line

A good voice AI knowledge base is mostly an editing job, not a technical one. Put behaviour and the fifteen common answers in the prompt, put the documented long tail in retrieval, put anything current behind a tool call, rewrite every document to be heard rather than read, and give the agent explicit permission to not know things. Then watch the transcripts, because the questions your agent fumbled last week are the exact list of what to write next.

Want your client watching those transcripts in a portal with your logo on your domain? Start free on VoiceDash or book a demo and I will walk you through the setup on a call.