RAG (retrieval-augmented generation): how AI answers from documents

Retrieval-augmented generation (RAG) is an approach where, before answering, the AI searches for relevant information in documents, a knowledge base or the internet and uses what it finds as the basis for its answer.

RAG is an open-book exam

In plain words

A closed-book exam is a regular model: it answers from memory and can mix things up. An open-book exam is RAG: first you find the right page, then you answer. There are fewer mistakes, and you can say where you got the information.

How it works

Documents are split into chunks in advance and indexed for fast search.
When you ask a question, the system looks for the chunks closest in meaning.
The fragments it finds are added to the prompt along with your question.
The model writes its answer based on these fragments and often links to the sources.
If nothing relevant is found, a good system will say so.

How retrieval-augmented generation works

Where you'll see it

RAG runs in chatbots with web search, in assistants that answer from your files or email, in support bots for shops and banks, and in company knowledge bases. When a bot shows footnote links next to sentences, it is most likely RAG.

Does it matter when buying

Yes, if you need accurate answers with up-to-date information. Choose services that show their sources, and open them. RAG noticeably reduces hallucinations but does not remove them completely: the model can misunderstand the fragment it found.

Myths

Myth: RAG makes the model smarter. Reality: The model is the same; it is just given a cheat sheet with the right facts.
Myth: If there is a link, the answer is definitely correct. Reality: The model can misreport the source. Open the links.

FAQ

How is RAG different from fine-tuning?

RAG gives the model information at the moment of the request; fine-tuning changes the model itself.

Can I set up RAG over my own notes?

Yes, some services let you upload files and ask questions about them.

Why does RAG sometimes give an outdated answer?

If the document base has not been updated for a long time, the model relies on old data.