What you will learn
- Describe the steps of a RAG pipeline
- Explain what embeddings and vector search do
- List the main design choices: chunking, retrieval, prompting
- Name the common failure modes and how to evaluate a system
A large language model knows what was in its training data and nothing since. It does not know your product documentation, last week's policy change or your customer records, and when asked about them it may produce an answer that sounds right and is not. Retrieval-augmented generation (RAG) addresses that: before the model answers, the system looks up relevant passages in your own documents and gives them to the model together with the question.
The idea in one sentence: retrieve first, then generate from what was retrieved.
The two phases
Preparing the documents (done ahead of time):
- Collect and clean the source documents: web pages, PDFs, tickets, wiki pages.
- Split them into chunks, pieces of a few hundred words, so that each holds one idea.
- Create an embedding for each chunk. An embedding model turns text into a list of numbers such that texts with similar meaning end up close together.
- Store the embeddings with the text and its metadata (source, title, date, access rights) in a vector index or database.
Answering a question (done for each question):
- Embed the question with the same model.
- Retrieve the chunks whose embeddings are nearest to the question's. Many systems combine this with ordinary keyword search, because exact terms (an error code, a product name) are matched better that way.
- Optionally re-rank the candidates with a more precise model.
- Build a prompt that contains the question, the best chunks and instructions such as "answer only from the sources; say so if they do not contain the answer; cite the source".
- Generate the answer, and show the sources so the reader can check.
What makes it work or not
Chunking. Chunks that are too large dilute the meaning and waste the context window. Chunks that are too small lose the surrounding context. Splitting at natural boundaries (headings, paragraphs) and keeping a little overlap helps, and including the document title in each chunk is a cheap improvement.
Retrieval quality. If the right passage is not retrieved, no model can answer correctly. Retrieval is usually the weakest link, so test it on its own: for a set of questions, is the passage that contains the answer in the top five results?
The prompt. Tell the model to rely on the sources, to say when it does not know, and to quote or cite. Keep instructions and retrieved text clearly separated.
Freshness and permissions. Re-index when documents change. Store who may read each chunk and filter by it at retrieval time, so one user's question never surfaces another's confidential document.
How it goes wrong
| Failure | What you see | What helps |
|---|---|---|
| The answer was never retrieved | A fluent but wrong or evasive answer | Better chunking, hybrid search, re-ranking, a larger top-k |
| Retrieved text is irrelevant | The model builds an answer from the wrong passage | Filters by metadata, re-ranking, a minimum similarity |
| The model ignores the sources | An answer from its own memory, contradicting the documents | A stricter prompt, and checking answers against the cited text |
| Outdated or conflicting documents | Contradictory answers | Version and date the sources, retire old ones |
| Injected instructions in a document | The model follows text that was in the data, not from you | Treat retrieved text as untrusted, and limit what the model can do |
| Sensitive data exposed | Someone sees what they should not | Permission filtering at retrieval, and careful choice of what to index |
Evaluating it
Judge retrieval and answers separately, so you know where a failure comes from.
- Write a set of realistic questions, with the document and passage that answers each.
- Retrieval: how often is the right passage in the top results?
- Answers: is the answer correct, complete and supported by the cited text? Have people grade a sample, and be careful when using a model as a grader, because it has its own biases.
- Include questions the documents cannot answer. A good system says it does not know.
- Re-run the set whenever you change the chunking, the models or the prompt.
When RAG is the right tool
It suits question answering over a body of text that changes: documentation, policies, support history. It is a poor fit when the answer needs calculation across many records (use a database query), or when you need a specific writing style or behaviour (consider adjusting the prompt or, later, fine-tuning). Start small, with a few documents and honest tests, before building infrastructure around it.
Glossary
- Embedding
- A list of numbers that represents the meaning of a piece of text, so similar meanings are close together.
- Vector database
- A store that finds the stored embeddings nearest to a query embedding, quickly.
- Chunk
- A piece of a document, small enough to embed and to fit into a prompt.
- Context window
- The amount of text a model can read in one request.
- Grounding
- Making the model base its answer on supplied sources instead of only on what it learned in training.
- Hallucination
- A confident answer that is not supported by the sources or by fact.
Frequently asked questions
Is RAG better than fine-tuning?
They solve different problems. RAG supplies facts at question time and is easy to update. Fine-tuning changes how a model behaves or writes. For answering from changing documents, RAG is usually the first thing to try.
Does RAG stop hallucinations?
It reduces them by giving the model sources, but does not remove them. The model can still misread the sources, or answer when the sources do not contain the answer, so you have to design for that and test it.
Next steps
- Build a small pipeline over a handful of your own documents
- Write 30 realistic questions with known answers and measure retrieval and answers separately
References
From Grokem: https://www.grokem.com/ai-and-machine-learning/generative-ai/large-language-models/retrieval-augmented-generation/retrieval-augmented-generation-explained
