RAG is not a knowledge strategy.

Retrieval-Augmented Generation is genuinely useful. It lets language models answer questions grounded in specific documents, rather than relying solely on training data. For many enterprise use cases, it's the right architectural choice.

It is not, however, a knowledge strategy. And the confusion between the two is causing a lot of quietly failing AI projects.

Garbage in, garbage out — with confidence

A RAG system retrieves content from a document store and uses it to generate a response. The quality of that response depends entirely on the quality of what's in the store.

If your documents are outdated, internally contradictory, poorly written, or structured for humans rather than retrieval systems, the model will do its best with what it has. Its best will be confidently wrong.

This is worse than a system that says "I don't know." A confident wrong answer, sourced from an internal document, feels authoritative. People act on it.

Before you think about vectors and embeddings, ask: is the content itself actually accurate, current and authoritative?

The chunking problem

RAG systems work by breaking documents into chunks, embedding those chunks, and retrieving the most relevant ones at query time.

A chunk that makes sense in the context of a full document often makes no sense in isolation. A paragraph that begins "As described in the previous section..." tells a retrieval system nothing useful. A table without its header row is meaningless. A sentence that references "the policy" without specifying which policy creates ambiguity that the model will resolve — incorrectly — by guessing.

Good RAG requires good chunking. Good chunking requires understanding how your content is actually structured, and restructuring it where necessary.

Outdated and contradictory sources

Most organisations have documents that contradict each other. A policy updated in 2023 that never formally replaced the 2019 version. A product spec that reflects the original design, not what was actually built. Meeting notes that contain decisions that were later reversed.

A language model retrieving from this corpus will surface contradictions without understanding they're contradictions. It will average them out, pick the more recent one, or blend them — depending on what the retrieval returns.

The result is answers that seem reasonable and are subtly wrong.

Content governance is not optional in a RAG architecture. Someone has to own the document corpus. Someone has to maintain it. Someone has to decide what's authoritative.

Access control and confidentiality

Not all documents should be accessible to all users. HR policies, compensation data, legal communications, customer-specific contracts — these exist in most organisations, and they end up in the knowledge base because someone thought "the AI should know everything."

A RAG system without access control will retrieve and surface content to users who shouldn't see it. Not always. Not obviously. But at some point, someone will ask a question that returns an answer they shouldn't have.

Build access control into the architecture from the beginning. It's significantly harder to retrofit.

Retrieval quality vs. answer quality

These are different things and they fail independently.

Retrieval quality: did the system find the right documents? This can be evaluated separately, with known queries and known relevant sources.

Answer quality: given the retrieved documents, did the model produce a good answer? This requires evaluating the generated text.

A system can retrieve perfectly and still generate a poor answer. It can retrieve poorly and — by luck — still generate something acceptable.

Evaluating only the final answer tells you less than you think. Evaluating the retrieval step separately gives you much more actionable signal about what's actually going wrong.

When the system should say "I don't know"

A knowledge system that answers everything is not trustworthy.

If a user asks a question that isn't covered in the document corpus, the honest answer is "I don't have reliable information about that." This requires the system to have a sense of its own coverage — what it knows, and what it doesn't.

Building this requires more than retrieval. It requires confidence thresholds, fallback behaviour, and a clear decision about what happens when retrieval returns nothing useful.

Organisations that skip this step end up with systems that answer confidently on topics where they have no reliable source. Users learn not to trust the system. The project is quietly abandoned.

RAG is an architectural pattern. Knowledge strategy is an organisational capability.

You can have excellent RAG infrastructure and terrible knowledge management. The infrastructure will faithfully retrieve and surface whatever you've put into it.

What you put into it is the actual problem.