RAG, explained for people who run a business

You will see the term RAG, retrieval-augmented generation, in every AI proposal you receive this year. It is simpler than it sounds, and the important part has nothing to do with AI.

The problem it solves

A language model on its own knows what it was trained on, which is the public internet up to a date. It does not know your client files, your pricing, your procedures or what you agreed with a supplier in March. Ask it about those and it will either say so or, worse, make something plausible up.

What RAG does

Before the model answers, the system searches your own information for the passages most relevant to the question, and hands those passages to the model along with the question. The model answers from what it was given and, if built properly, cites which document each part came from.

The "search" step usually works on meaning rather than keywords, using a vector database, which is why it can find "the note about the Henderson renewal" when you ask "what did we say about extending the Henderson contract".

So: retrieve first, then generate. The model never needs to be trained on your data, and your documents stay where they are.

The part that matters: who may see what

Retrieval is a permissions problem before it is an AI problem. If the search index contains every document in the firm, the assistant will happily surface the partners' salary spreadsheet to whoever asks the right question. The system must retrieve only what the person asking is entitled to see, using the same identity and permissions your file systems already use.

This is why "switch on an AI assistant" usually starts with a less glamorous job: tidying up sharing settings, group membership and the folders nobody has reviewed since 2019.

Where the index lives

The vector index is a copy of your information in searchable form, so where it lives is a data-governance decision. For sensitive data it belongs inside your own environment: your tenant, your cloud subscription, your region, your keys, your logs. For public marketing content, it matters much less. Match the architecture to the classification, not to the vendor's default.

Questions to ask any RAG proposal

• Which sources are indexed, and who decided?
• How does retrieval respect my existing permissions?
• Where does the index live, and who can reach it?
• Does every answer cite its sources?
• What is logged, and where?

If the answers are vague, the proposal is a demo, not a system.