A Knowledge Base Chatbot Without a Vendor Cloud: Grounding an Assistant in Your Own Articles

Monday, 11:05, at Fenwick & Hale, a firm of surveyors with ninety staff and a two-person desk. A partner has forwarded a vendor e-mail that promises "an AI chatbot trained on your knowledge base" that will "deflect sixty per cent of tickets", and has asked Ruth, who runs the desk, whether that is true and whether the firm's client files would end up "in the AI". Ruth has sixty-one knowledge articles, nineteen of which are current, a help desk tool with an assistant feature she has never switched on, and a security policy that says client data stays on the firm's servers. She has a week to answer. The vendor's e-mail answered neither of the partner's questions, and this article is the answer Ruth wrote.

A knowledge base chatbot is an assistant that answers a requester's question by finding the relevant articles in the desk's knowledge base and writing a reply from them, rather than from whatever the underlying model learned elsewhere. The finding part is called retrieval; the writing-from-what-was-found part is called grounding, and the two together are what vendors mean by a bot "trained on your knowledge base", although nothing is trained. The same machinery is sold as "AI-powered knowledge base software" when the vendor leads with the articles and as a "chatbot" when it leads with the chat window; it is one thing, and the questions to ask it are the same. The idea is sound and the word "chatbot" has been attached to enough bad products that it needs taking apart. This article is part of our AI Assistants in Support series, which starts with What an Assistant Can and Cannot Do.

It covers what grounding means and why it is the whole point. Then come the four places the data can go, in a table, because that is the partner's second question and the one the vendor did not answer. It covers what a grounded assistant may answer and what it must not, the articles it needs from you before it is worth switching on, and how to tell in a month whether it helped. There is a short war story. It ends with a checklist to run before the bot talks to a single requester.

What Grounding Means

A language model on its own answers from what it absorbed while being built: a great deal of the public internet, which includes several thousand pages about printers and none about yours. Asked "why won't the third-floor printer scan to e-mail?", it will produce a confident, general answer, because producing confident general answers is what it does. That answer is a guess in a good suit, and the first time it is wrong about your printer the requester stops trusting the bot and, shortly afterwards, the desk.

Grounding changes the question the model is asked. Instead of "why won't the printer scan?", the system first searches the knowledge base for articles about the third-floor printer and scanning, and then asks the model: "using only these three articles, answer this question, and if they do not contain the answer, say so." The model's job shrinks from knowing to reading. It can still misread, and it will sometimes blend two articles into one wrong instruction, which is why the article that covers the pattern at the desk is Drafting Replies With a Human Reviewer. But the ceiling on how wrong it can be is now set by your articles, not by the internet.

Three consequences follow, and they decide whether the bot is worth having. The bot is exactly as good as the articles it can find, so a knowledge base with nineteen current articles out of sixty-one gives a bot that is wrong two times in three. The bot must be able to say "I do not know; here is how to raise a ticket", and a bot that cannot is not grounded, whatever the brochure says. And the retrieval step must respect who is asking: an article that only agents may see must not be found for a requester, which is a question to ask every vendor and the first thing to test.

Where the Data Goes: Four Shapes

The partner's question, "would the client files end up in the AI?", has four possible answers depending on how the thing is built. They differ in what leaves the building, what it costs, and who can switch it off. The table is the four, with the honest answer for each.

Shape 4 is not a consolation prize. A portal that shows the three most relevant articles while the requester is still typing the subject line deflects a surprising share of the simple tickets, with no model, no cost and no wrong answers, because it does not answer; it points. Suggesting Knowledge Articles is the article on doing this for agents; the same mechanism on the requester's side is the first thing to switch on, and the thing to measure the bot against.

Shapes 2 and 3 are the ones a self-hosting organization can live with. In shape 2 the question and the articles it retrieved do leave the building, to a provider the firm chose and signed terms with, which is a different thing from "the AI" in general and a thing the security policy can take a position on. In shape 3 nothing leaves, at the price of running a model yourself: a modern small model on a server with a reasonable graphics card answers a knowledge base question in a few seconds, which is fine for a bot and slow for a chat. Some help desk tools, QuerySolv among them, let you choose between shapes 2 and 3 with your own provider and key or a local endpoint, meter nothing, and keep the "what leaves the building" list on one page. Ask any vendor for that page. If there is not one, assume shape 1.

The ticket history is the detail to press on. A bot that answers a requester's question needs the articles, not the tickets. A bot that drafts an agent's reply to an open ticket needs that ticket's history, which may contain names, attachments and the thing the requester said in confidence. Shape 2 sends that history to the provider; shape 3 keeps it. The two uses can be configured differently, and should be. Guardrails: What Leaves the Building is the article on drawing that line.

What It May Answer, and What It Must Not

A grounded assistant is good at exactly one kind of question: the one whose answer is in an article and is the same for everyone who asks. How to connect to the Wi-Fi, how to book a meeting room, what the VPN client is called, where the expenses form lives, why scanning to e-mail needs the full address. These are the first twenty articles any desk writes, and The First Twenty Articles is where they are listed. For these, a bot that quotes the article, links it and offers a ticket if it did not help is a better first contact than a form, because it is faster and it works at 23:00.

It must not answer four kinds, and the configuration should make it unable to rather than merely unlikely to. Anything about a specific person's account, because "is my account locked?" needs a lookup the bot should not have and an identity check it cannot do. Anything that requires an action: resets, access, orders, approvals, which are tickets with owners and clocks. Anything where the honest answer is "it depends on your situation", because the bot will pick one situation. And anything the knowledge base does not cover, where the only right reply is "I could not find this; here is how to raise a ticket and the desk's hours". The list of what to keep away from an assistant, with the reasoning, is in What Not to Hand Over to an AI Assistant.

The handoff is the design decision that matters most. A bot whose "raise a ticket" path is three clicks away has been designed to reduce tickets rather than to answer questions, and requesters notice the difference in about a day. The right shape is one click, carrying the question the requester already typed into the ticket as its description, so that the desk receives a ticket that is better than most hand-written ones. Deflection that is really obstruction shows up in the numbers within a month, as fewer tickets and more walk-ups, and Ticket Deflection Strategy is the series on telling the two apart.

The Articles It Needs From You

Ruth's sixty-one articles were the real project, and the bot was the reason to do it. A knowledge base that serves a bot needs four properties that a knowledge base for humans can survive without. One question per article, because retrieval finds articles, and an article that answers six questions is found for all six and quoted for the wrong one. The question in the title, in the requester's words, because that is what the retrieval matches against; "Scan to e-mail fails on the Canon" is found, "MFD configuration notes" is not. A date and an owner on every article, because a bot that quotes last year's VPN instructions does so with perfect confidence; the method for keeping the dates honest without a documentation day is in Maintenance Without a Documentation Day. And a hard line between public and internal articles, enforced by the tool, because the internal article has the admin password step in it and the bot must never see it on the requester's behalf.

The practical order is to run the health check in A Knowledge Base Health Check first, retire or fix what it finds, and only then switch the bot on over the articles that survived. Ruth's nineteen became twenty-eight in a fortnight, all dated, all titled as questions, and the bot went live over those twenty-eight and nothing else. A bot over sixty-one articles of which forty-two are stale is a bot that will be switched off by Thursday, and rightly.

The bot then becomes the best article-writing tool the desk has ever had, for a reason nobody advertises: it logs the questions it could not answer. That log is the list of the next articles to write, in the requesters' own words, ranked by how often they asked. Capture Knowledge While Solving is the series on turning that list into articles without a writing day.

Measuring Whether It Helps

Four numbers, read monthly, against the month before the bot. Tickets per hundred staff, which should fall for the categories the articles cover and nowhere else; a fall everywhere means people stopped asking, which is not the same thing. The bot's "could not answer" rate, which should start high and fall as the articles grow; a rate that stays high means retrieval is failing or the articles are not written as questions. The share of bot conversations that became a ticket through the handoff, with the ticket's description already filled, which is the bot doing its honest job. And reopens plus walk-ups, which rise when the bot is wrong often enough that people route around it. A bot that improves the first and second without worsening the fourth is earning its place. One that improves the first by worsening the fourth is obstruction with a friendly face.

Read the transcripts too, twenty a month, chosen at random. Vendors show the ones that went well. The twenty will show the question the bot answered confidently from the wrong article, which is the kind of error no number catches and the kind that costs the most trust. The fix is nearly always an article, not a setting.

A Short War Story: The VPN Article at Fenwick & Hale

The bot went live on a Tuesday over twenty-eight articles, in shape 2, with the firm's own key to a hosted provider and the ticket history kept out of it. On Wednesday a surveyor working from a client's site asked it how to connect to the VPN, and the bot answered correctly from the article, except that the article was the one Ruth had rewritten the week before and the surveyor's laptop still had the old client installed. The bot's answer named the new client, the surveyor could not find it, and the conversation ended in a ticket, raised through the one-click handoff with the whole exchange attached. Ruth read it at 09:10, saw in one glance that the article was right and the laptop was old, and added two lines to the article: how to tell which client you have, and what to do if it is the old one. The next three surveyors with old laptops got the two lines and did not raise a ticket. The partner, who had been sent the transcript, asked whether the client files had been involved. Ruth showed him the provider agreement, the one-page list of what left the building, and the setting that kept ticket histories out of it. He asked why the vendor's e-mail had not said any of this. Ruth said that it had answered a question about the bot, and he had asked a question about the firm.

The Checklist Before It Talks to Anyone

The checklist is the twelve things to confirm before the bot answers its first requester. Most of them are about the knowledge base and the configuration, not the model, which is the point. Run it again whenever the tool, the provider or the knowledge base owner changes.

KNOWLEDGE BASE CHATBOT — GO-LIVE CHECKLIST
Desk: ______________   Shape (1-4): ___   Provider or model: ______________   Owner: __________

THE ARTICLES
 [ ] Every article the bot can see answers one question, with the question in the title.
 [ ] Every article has a date and an owner; nothing older than the review period is live.
 [ ] Internal articles are marked internal in the tool, and the bot cannot retrieve them
     for a requester. Tested with a question only an internal article answers.

THE DATA
 [ ] The one-page list of what leaves the building exists, and names the provider.
 [ ] Ticket histories are not sent for requester-facing answers (and the setting is yours).
 [ ] The provider agreement has been read by whoever owns the security policy.

THE ANSWERS
 [ ] The bot says "I could not find this" when the articles do not cover the question.
     Tested with five questions that are not in the knowledge base.
 [ ] The bot refuses account-specific, action and it-depends questions with the handoff.
 [ ] The handoff is one click and carries the requester's question into the ticket.

THE MEASURES
 [ ] Baseline taken: tickets per 100 staff by category, reopens, walk-ups, for last month.
 [ ] The "could not answer" log is visible to the knowledge base owner.
 [ ] Twenty transcripts a month are read by a person, and the findings go into articles.

Ruth's answer to the partner was a page: shape 2, the provider named, the list of what left, the ticket histories kept in, the twenty-eight articles, and the four numbers she would report in a month. The vendor's sixty per cent did not appear on it. The firm's own figure, three months on, was that tickets in the eight covered categories had fallen by about a third and walk-ups had not risen, which was enough, and which was true.

Frequently Asked Questions