RAG explained: how to make enterprise knowledge usable for AI

Your organisation holds knowledge that exists in no language model: contracts, project reports, product documentation, internal policies, email threads. When employees ask ChatGPT or another AI tool about internal topics, they receive a refusal at best. At worst, a plausible-sounding but incorrect answer. Retrieval-Augmented Generation, or RAG for short, closes this gap. The approach connects a language model with your own documents, without the model having to be trained for it. This guide explains how it works, why source citations are the real gain in trust, and why most RAG projects fail not because of the AI, but because of access rights.

What is Retrieval-Augmented Generation?

Retrieval-Augmented Generation (RAG) is an approach in which a language model, before answering a question, receives relevant text passages from a defined document collection and bases its answer on those passages. The name describes the two steps: retrieval (fetching matching content) and generation (the model formulating the answer). The language model itself remains unchanged. It does not draw on hard-wired training knowledge, but on the documents the system supplies to it at runtime.

An analogy from everyday work: a new employee does not answer a specialist question from memory, but looks it up in the manual and summarises the passage they find. RAG does exactly that, only in seconds and across the entire body of documents.

How RAG works technically – in four steps

For decision-makers, an understanding at the process level is sufficient. The details are handled by a RAG developer. The steps are nevertheless important for assessing effort and risks.

• 1. Preparation: Your documents (PDFs, wiki pages, tickets, contracts) are broken down into manageable sections. This step is called chunking. How well this segmentation succeeds later co-determines the quality of the answers.
• 2. Indexing: Each section is converted into a numerical representation, a so-called embedding. Embeddings represent the meaning of a text as a sequence of numbers, so that texts with similar content lie computationally close to one another. This is stored in a vector database, a database specialised in similarity search.
• 3. Retrieval: When a user asks a question, the system pulls out the sections that come closest in meaning to the question. Even when the wording differs. Whoever asks about a “notice period” will also find documents that speak of “contract termination”.
• 4. Answer: The sections found are handed to the language model together with the question. The model formulates an answer that is based on precisely these sections, and can state which statement comes from which document.

Why no training is necessary

A widespread misconception goes: “We first have to train the AI with our data.” For the vast majority of knowledge use cases, this is not true. Training a language model, or adapting it after the fact, so-called fine-tuning, changes the weights of the model itself. This is expensive, requires specialist knowledge, and must be repeated with every change in knowledge. Above all, however, it solves the wrong problem. Fine-tuning is suitable for teaching a model a style or a behaviour, not for making facts reliably retrievable.

RAG takes the opposite approach: the knowledge stays outside the model, in your systems, and is only drawn upon when needed. The differences at a glance:

From a data protection perspective, too, the difference is relevant: with RAG, a document can be removed from the index, and it is gone. Knowledge that has flowed into a model via fine-tuning can practically no longer be deleted in a targeted way. A problem as soon as data subject rights under the GDPR come into play.

Source citations: the underestimated advantage

Language models can invent statements that sound plausible but are false. The phenomenon is referred to as hallucination. For enterprise use, this is the central destroyer of trust. An answer that no one can verify is worthless in many contexts.

RAG defuses the problem on two levels. First, the tendency to hallucinate decreases when the model is instructed to answer only on the basis of the supplied passages. Second, and this matters more in practice, the system can support every statement with its source: document name, section, ideally a direct link into the source system. Employees verify the answer in seconds, instead of having to trust it blindly.

To be honest, it must also be said: RAG does not eliminate hallucinations entirely. A model can summarise retrieved passages incorrectly or link statements that do not belong together. Source citations, however, make such errors traceable, and that is the qualitative difference from a model without source references. For applications with liability relevance (legal advice, medical statements, binding price commitments), human review therefore always belongs in the process.

Access rights: the stumbling block where projects fail

The technology behind RAG is by now well understood. What projects actually fail at in practice is usually something else: permissions.

The problem arises like this: a RAG system typically indexes documents from several sources, for instance SharePoint, Confluence, file servers, CRM. Access rights apply in each of these systems. Only HR sees the salary list, only the management board sees the board report. If all of this is loaded into a shared index and the RAG system answers every user from the overall body of documents, these boundaries are removed. The system then becomes an unintended information desk for things the person asking should never have been allowed to see.

The solution is called permission-aware retrieval. With each request, the system checks which documents the specific user would be allowed to see in the source system, and draws on only these for the answer. That sounds obvious, but it is the most labour-intensive part of many RAG projects. There are three reasons for this:

• Permissions must stay in sync. When someone changes department or leaves the company, this must take effect promptly in the RAG system, not only at the next rebuild of the index.
• Source systems speak different languages. SharePoint groups, Confluence spaces and file server ACLs (Access Control Lists) follow different logics that have to be mapped onto a common model.
• Permissions that have grown over time are often messy. Many companies discover during the RAG project that their existing shares are scoped too broadly. That is not an AI problem. But the RAG project makes it visible and should not make it worse.

Our market assessment: whoever considers access rights only after the pilot will build the pilot twice. The permissions question belongs in the first project week, not the last.

What RAG cannot do

A realistic set of expectations is part of any serious roll-out. You should be aware of three limits:

• RAG does not repair poor documentation. Outdated, contradictory or incomplete documents lead to outdated, contradictory or incomplete answers. The system makes the quality of knowledge visible, it does not produce it.
• RAG is poor at calculation and aggregation. Questions such as “How has our revenue developed by region?” require structured analyses across databases, not a text search. Other architectures are responsible for that, which can be combined with RAG.
• RAG does not understand implicit connections across many documents. Answers that require bringing together knowledge from dozens of sources overwhelm the classic retrieval mechanism. Here, advanced approaches help, which an experienced developer selects depending on the use case.

How to start a RAG project sensibly

A proven approach for getting started:

• Choose one use case, not ten. Well suited: a clearly delimited body of documents with high demand, for example product documentation for support or internal policies for HR enquiries.
• Clarify the permissions situation in advance. Which source systems, which rights models, who may see what? These answers determine the architecture.
• Decide on the operating model. Cloud models in EU regions, dedicated instances or locally operated models: the choice depends on protection requirements and budget and should be coordinated with data protection officers.
• Measure quality, do not guess it. Before the roll-out, define a test set of real questions and evaluate the answers systematically. Without measurement, it is impossible to judge whether the system is ready for production.
• Time-box the pilot. A working pilot is typically achievable within a few weeks. The path to productive operation takes longer, because permissions and data quality make up the main effort.

There are two routes for implementation. You bring expertise into your team: a vetted RAG developer brings the experience from comparable projects and avoids the typical beginner mistakes with chunking, retrieval quality and permissions. Or you opt for a ready-designed solution such as our Knowledge Agent, which implements the described setup as a package. Terms you encounter in conversations with service providers are explained by our glossary.

Conclusion

RAG is the pragmatic way to make enterprise knowledge usable for AI: no training, up-to-date answers, traceable sources, documents remain under your control. The technology is mature. The success of the project is decided by data quality and access rights. Our recommendation for action: start with a single, clearly delimited use case, clarify the permissions question before the first line of code, and measure answer quality before you roll out. If you need reinforcement for this, you will find vetted RAG developers through us. Or check whether the Knowledge Agent already covers your case.