RAG explained: AI that answers with your own business data

A colleague asks the chatbot on your intranet what the delivery time is for a custom order. The answer comes back fast, sounds confident, and is wrong. It says "five working days", while your own terms say three weeks. The chatbot has never seen those terms. It just came up with something plausible.

That is exactly the problem RAG solves. With RAG you get AI to answer with your own data: not from whatever the model once read on the internet, but from your documents, your terms, your manuals. In this piece we explain how that works without the jargon, what it is good for, and where it tends to go wrong in practice.

Why a language model does not know your business

A language model like the one behind ChatGPT was trained on huge amounts of text. It knows a lot about the world in general. About your business, it knows nothing. Your price list, your returns policy, the agreements with your biggest supplier, none of it is in there.

Ask the model about them anyway and it does what it always does: it predicts which text logically comes next. Sometimes that happens to be right. Often it is not. And the annoying part for you as a user: a wrong answer sounds just as convincing as a right one. No warning light goes on.

You cannot simply retrain the model on your own documents. That is expensive and slow, and every time your terms change you fall behind again. There is an easier route.

How RAG works, in plain language

RAG stands for retrieval-augmented generation. Put simply: look it up first, answer second.

Think of a new hire on your customer service desk. They do not know everything by heart yet. But if they check the right folder before every answer, they give good answers. RAG does exactly that, only very fast.

Behind the scenes, a few steps happen:

1. Your documents get split up. Manuals, FAQs, terms, product information. Each document is divided into smaller pieces, for example per paragraph or per topic.
2. Those pieces are made searchable. Not just on literal words, but on meaning. If someone asks about "sending something back", the system also finds the piece about "returns".
3. When a question comes in, the relevant pieces are retrieved. The system finds the few chunks of text that best match the question.
4. The model gets the question plus those pieces. With the instruction: answer only based on this information.
5. The answer comes back with a source. So you or your customer can see where it came from.

So the model itself does not change. You hand it the right information with every question. Update a document, and from that moment the system uses the new version.

What RAG is good for in practice

RAG works best where people keep asking about information that is already written down somewhere. A few places we see it a lot:

Customer service

Questions about delivery times, warranty, returns, installation. The answers sit in your documentation, but customers do not read it. A chatbot with RAG looks up the right piece and answers in plain language. Whether you use an off-the-shelf tool for that or have something custom built, we lay out in AI chatbot for your business: ChatGPT wrapper or custom?.

Internal knowledge base

Every business has knowledge spread across shared folders, wikis, and the head of that one colleague who has been there for ten years. With RAG, a new employee can simply ask how to create a credit note or what steps to take during an outage, and get the answer with a link to the right document.

Finding quotes and contracts

"What did we agree with this customer about maintenance last year?" Searching through hundreds of PDFs is slow. A RAG system finds the passage and summarizes it. Important: the source stays attached, so someone can check it before anything goes to the customer.

Where it goes wrong

RAG sounds simple, and the principle is. The pitfalls are in the execution. These are the places where we most often see it stumble in projects.

Bad or outdated sources. RAG is only as good as the documents you feed it. If your folder holds three versions of your terms, two of them outdated, the system may grab the wrong one. Cleaning up your sources is often more work than the technology itself, and it is work you cannot skip.

Documents split in awkward places. Cut a text in the wrong spot and the context falls away. A price table cut in half produces an answer with half the information. How you split depends on the type of document. A manual needs something different from a contract.

The search step misses the right piece. If your customer uses different words from your documentation, or the answer is spread across three documents, the system sometimes retrieves the wrong pieces. The model then answers neatly, but based on the wrong information.

No "I don't know". A good system honestly says when it cannot find the answer. Without that instruction, and without testing whether it sticks to it, the model fills the gaps itself after all. Then you are back to the chatbot from the first paragraph.

Permissions. Not everyone gets to see everything. Put salary information and customer contracts in the same knowledge base as your product information, and the system needs to know who may retrieve which documents. You handle that at the retrieval step, not afterwards in the answer.

Nobody tests in a structured way. Trying a few questions and thinking "works fine" is not a test. You need a fixed set of questions with the answers you expect, which you rerun after every change.

How to test whether it works

Start with a list of around thirty to fifty real questions. Pull them from your inbox, your customer service, or your colleagues. For each question, write down the correct answer and which document it is in.

Have the system answer those questions and look at three things: did it find the right document, is the answer correct, and does it honestly say "I don't know" when the answer is not in there. Keep that list. Whenever you change something, in the documents, the splitting, or the instructions, run it again. That way you see straight away whether you improved something or broke it.

What it costs and how to start

A first working version on a defined set of documents is usually up in 2 to 4 weeks. That lets you learn quickly whether your sources are good enough and whether people actually use it. A production-ready system with integrations, permissions, and monitoring is a bigger project. What that roughly costs, and which items count toward it, you can read in what it costs to build an AI agent.

Want AI not just to look things up in documents, but also to pull data from your existing systems, like your CRM or planning tool? Then you get to integrations through a standard like MCP. We wrote about that in MCP: how to connect AI to the software you already use. And if your business already runs on its own web application, RAG can often be added as a feature inside it.

Our advice: start small. Pick one topic that draws a lot of questions, clean up those documents, and test with real questions. If that works, expand.

Curious whether your documents are a good fit?

Sitting on a folder full of manuals nobody reads, or a customer service team answering the same questions over and over? We are happy to look at whether RAG solves something there, and whether it is worth it. Sometimes the answer is: sort out your FAQ first. Book a call, and we will go through it together.

Frequently asked questions

What exactly does RAG mean?

RAG stands for retrieval-augmented generation. The system first looks up the relevant pieces of your documents and passes them, together with the question, to a language model. The model then answers based on your information, instead of on what it knows in general.

Do I need to train my own AI model to use AI with my own data?

No. With RAG the model stays as it is. You hand it the right pieces of your documents with every question. That is faster and cheaper than training, and when you update a document, the system uses the new version right away.

Can RAG still make things up?

Yes, if it is not set up well. If the system retrieves the wrong pieces, or there is no instruction to say "I don't know", the model can fill the gaps itself. That is why you test with a fixed set of real questions and always show the answer with its source.

Is my business data safe with RAG?

That depends on how you build it. Pay attention to where your documents are stored, which AI service processes the questions and under what terms, and who may retrieve which documents. You handle those permissions at the retrieval step, so an answer never shows something a person was not allowed to see.