RAG (Retrieval-Augmented Generation) Explained: Chunking, Embeddings, Hybrid Search and a Working Pipeline

Ask an LLM about your institute's refund policy and it will answer fluently and confidently, and it will make something up. It was never shown that document. Its knowledge stops at a cutoff date, it knows nothing private, and it will happily hallucinate to fill gaps (Session 21).

You could retrain the model on your documents, but that is slow, costly and goes stale. The much simpler fix is retrieval-augmented generation (RAG): when a question arrives, first look up the relevant passages from your own documents, put them in the prompt, and tell the model to answer only from them. The model stays the same. Only the context changes. The idea comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, and it is now the standard way to build chatbots over private data.

In this session we build a real pipeline: chunk, embed, store, search, and answer with a local model. And we test it honestly, including where keyword search beats embeddings and where it does not.

The pipeline in two phases

Phase A, indexing (offline, once): load documents, clean them, split into chunks, turn each chunk into a vector with an embedding model, and store the vectors with the text and metadata.

Phase B, query time (online, every question):

1. Embed the question with the same embedding model.
2. Retrieve the top-$k$ most similar chunks.
3. Augment: put them in the prompt with the question and a rule such as "answer only from the context".
4. Generate: the LLM writes the answer, ideally with citations.

Why bother, compared with the alternatives?

RAG gives fresh knowledge without retraining, source citations, fewer hallucinations and per-user access control. Often RAG and fine-tuning are combined.

Our knowledge base

To test honestly we need text the model has never seen. Here is a made-up tuition centre's policies. The code in the rest of the session starts from DOCS:

DOCS = [
 "Refund policy: fees are refundable within 7 days of enrolment if fewer than 3 classes were attended. After that, no refund is given.",
 "Class timings: Pre-medical batches run Monday to Friday, 4 pm to 7 pm. Pre-engineering batches run 2 pm to 5 pm.",
 "Monthly fee: the pre-medical course costs Rs. 9,500 per month and the pre-engineering course costs Rs. 8,000 per month.",
 "Scholarship: students who score above 85 percent in Matric receive a 20 percent fee discount for the first year.",
 "Error ERR-4012 in the student portal means the roll number was entered with letters. Re-enter it using digits only.",
 "Mock tests are held on the first Saturday of every month. Results are published on the portal within 48 hours.",
 "Contact: the front desk is open 9 am to 8 pm. Call 042-111-000-123 or visit the Gulberg branch.",
 "Transfer between batches is allowed once per term and requires written approval from the academic coordinator.",
 "Absence rule: a student who misses more than 6 classes in a month is called in for a meeting with a parent or guardian.",
 "Library: the library is open to enrolled students on weekdays and has past papers for MDCAT and ECAT.",
]

Chunking: how to cut documents

Real documents are long, and you cannot embed a whole book as one vector or paste it into a prompt. A chunk is a piece of a document (often 300 to 800 tokens) that is embedded and retrieved on its own. Too small and it loses context. Too large and it adds noise and cost.

import re

text = ("Student handbook. Attendance matters. Students must attend at least 80 percent of classes. "
        "Illness needs a doctor's note within three days. Fees are due on the 5th of every month; a late fee of Rs. 500 applies after the 10th. "
        "Mock tests count toward the final grade. A missed mock test can be retaken only once, with permission. "
        "Lockers are assigned at enrolment and must be emptied at the end of each term.")

def fixed(t, size):
    return [t[i:i + size] for i in range(0, len(t), size)]

def with_overlap(t, size, overlap):
    step = size - overlap
    return [t[i:i + size] for i in range(0, len(t), step) if t[i:i + size]]

def by_sentence(t, max_chars):
    chunks, cur = [], ""
    for s in re.split(r"(?<=[.;])\s+", t):
        if len(cur) + len(s) > max_chars and cur:
            chunks.append(cur.strip()); cur = ""
        cur += s + " "
    return chunks + [cur.strip()]

for name, chunks in [("fixed 120 characters", fixed(text, 120)),
                     ("120 with 30 overlap", with_overlap(text, 120, 30)),
                     ("whole sentences, max 160", by_sentence(text, 160))]:
    print(f"{name:26s} -> {len(chunks)} chunks")

print("\nfixed-size chunks 2 and 3 (notice where the cut falls):")
for c in fixed(text, 120)[1:3]:
    print("  |", c)
print("\nsentence-aware chunks:")
for c in by_sentence(text, 160)[:3]:
    print("  |", c)

fixed 120 characters       -> 4 chunks
120 with 30 overlap        -> 5 chunks
whole sentences, max 160   -> 3 chunks

fixed-size chunks 2 and 3 (notice where the cut falls):
  |  within three days. Fees are due on the 5th of every month; a late fee of Rs. 500 applies after the 10th. Mock tests cou
  | nt toward the final grade. A missed mock test can be retaken only once, with permission. Lockers are assigned at enrolme

sentence-aware chunks:
  | Student handbook. Attendance matters. Students must attend at least 80 percent of classes. Illness needs a doctor's note within three days.
  | Fees are due on the 5th of every month; a late fee of Rs. 500 applies after the 10th. Mock tests count toward the final grade.
  | A missed mock test can be retaken only once, with permission. Lockers are assigned at enrolment and must be emptied at the end of each term.

The same short handbook gives 4 fixed chunks, 5 overlapping ones and 3 sentence-aware ones. Look at where the fixed cut lands: chunk 3 begins with "nt toward the final grade", slicing the word count in half. A chunk that starts mid-word embeds poorly and reads badly. The sentence-aware chunks keep whole thoughts together, so each one is a clean unit for retrieval. Libraries such as LangChain's RecursiveCharacterTextSplitter do the sentence-aware version for you:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_text(document_text)

Embeddings and similarity search

An embedding model turns a piece of text into a vector so that texts with similar meaning land close together. This is a separate model from the LLM, and its output is one vector for the whole text. "Close" is usually measured by cosine similarity, the angle between two vectors:

A vector database stores the vectors and quickly finds the nearest ones (approximate nearest-neighbour search). Options range from purpose-built databases (Pinecone, Weaviate, Qdrant, Milvus, Chroma), to extensions of databases you already have (pgvector for PostgreSQL, MongoDB Atlas, Redis, Elasticsearch), to libraries (FAISS, Annoy, HNSWlib) and embedded stores (LanceDB, SQLite-vec). We use FAISS, Meta's library, because it needs nothing but Python.

import faiss
import numpy as np
from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("all-MiniLM-L6-v2")                       # a small embedding model (384 numbers per text)
vecs = embedder.encode(DOCS, normalize_embeddings=True).astype("float32")
print("embedded", len(DOCS), "documents into vectors of shape", vecs.shape)

index = faiss.IndexFlatIP(vecs.shape[1])                                  # inner product on unit vectors = cosine similarity
index.add(vecs)

def search(query, k=2):
    q = embedder.encode([query], normalize_embeddings=True).astype("float32")
    scores, ids = index.search(q, k)
    return [(DOCS[i], float(s)) for s, i in zip(scores[0], ids[0])]

for query in ["How do I get my money back?", "When are the afternoon classes for medicine students?", "I want to talk to somebody"]:
    print(f"\nQ: {query}")
    for doc, score in search(query):
        print(f"   {score:.2f}  {doc[:78]}...")

embedded 10 documents into vectors of shape (10, 384)

Q: How do I get my money back?
   0.34  Refund policy: fees are refundable within 7 days of enrolment if fewer than 3 ...
   0.17  Contact: the front desk is open 9 am to 8 pm. Call 042-111-000-123 or visit th...

Q: When are the afternoon classes for medicine students?
   0.60  Class timings: Pre-medical batches run Monday to Friday, 4 pm to 7 pm. Pre-eng...
   0.40  Library: the library is open to enrolled students on weekdays and has past pap...

Q: I want to talk to somebody
   0.10  Contact: the front desk is open 9 am to 8 pm. Call 042-111-000-123 or visit th...
   0.10  Library: the library is open to enrolled students on weekdays and has past pap...

The embedding model, all-MiniLM-L6-v2, produces a 384-number vector for each text. Look at the first query: "How do I get my money back?" shares no important word with the refund policy, which says "refundable" and "fees". Yet the refund document ranks first, with a cosine score of 0.34. That is the power of searching by meaning. The second query, "afternoon classes for medicine students", also finds the right document (0.60). The third, "I want to talk to somebody", is a weak match everywhere (0.10, 0.10), a warning that the top result is not always a good one, and a good reason to set a minimum score below which you answer "I do not know".

Keyword search, and why hybrid helps

Dense retrieval understands meaning but can miss exact tokens such as an error code, a name or a number. Classic keyword search (BM25) matches exact words, weighting rare ones more heavily.

The two methods fail in different ways, so combining them (hybrid search) often works best. A simple, popular way to merge two rankings without comparing their scores is reciprocal rank fusion (RRF):

Let's test them on four queries, each with a known right answer:

import math, re
import numpy as np
from sentence_transformers import SentenceTransformer

tokenize = lambda s: re.findall(r"[a-z0-9\-]+", s.lower())
docs_tok = [tokenize(d) for d in DOCS]
avgdl = sum(map(len, docs_tok)) / len(docs_tok)

def bm25(query, k1=1.5, b=0.75):
    """Keyword search: rare words matter more (idf), repeated words help a little, long documents are discounted."""
    scores = np.zeros(len(DOCS))
    for term in tokenize(query):
        n = sum(term in d for d in docs_tok)
        if n == 0: continue
        idf = math.log(1 + (len(DOCS) - n + 0.5) / (n + 0.5))
        for i, d in enumerate(docs_tok):
            tf = d.count(term)
            scores[i] += idf * tf * (k1 + 1) / (tf + k1 * (1 - b + b * len(d) / avgdl))
    return scores

embedder = SentenceTransformer("all-MiniLM-L6-v2")
dvecs = embedder.encode(DOCS, normalize_embeddings=True)
dense = lambda q: dvecs @ embedder.encode([q], normalize_embeddings=True)[0]

def rank(scores): return list(np.argsort(-scores))
def rrf(*rankings, k=60):                                  # reciprocal rank fusion: combine rankings without comparing their scores
    fused = np.zeros(len(DOCS))
    for r in rankings:
        for pos, i in enumerate(r): fused[i] += 1 / (k + pos + 1)
    return fused

tests = [("reimbursement of tuition", 0), ("first Saturday", 5), ("digits only", 4), ("who approves moving to another batch", 7)]
print(f"{'query':38s} {'keyword':8s} {'dense':8s} {'hybrid':8s}")
wins = {"keyword": 0, "dense": 0, "hybrid": 0}
for query, answer in tests:
    k, d = bm25(query), dense(query)
    got = {"keyword": rank(k)[0], "dense": rank(d)[0], "hybrid": rank(rrf(rank(k), rank(d)))[0]}
    for name in wins: wins[name] += got[name] == answer
    print(f"{query:38s} " + " ".join(f"{'right' if got[n] == answer else 'WRONG':8s}" for n in wins))
print("\ncorrect out of", len(tests), ":", wins)

query                                  keyword  dense    hybrid  
reimbursement of tuition               WRONG    right    right   
first Saturday                         right    WRONG    right   
digits only                            right    WRONG    right   
who approves moving to another batch   WRONG    right    WRONG   

correct out of 4 : {'keyword': 2, 'dense': 2, 'hybrid': 3}

The result is realistic rather than flattering. Dense search wins on "reimbursement of tuition" (no shared word with "refund") and on the paraphrase "who approves moving to another batch". Keyword search wins on "first Saturday" and "digits only", where the exact words appear in the right document and the embedding model wandered off. Hybrid got three of four right, better than either alone (2 of 4 each), but it still missed one: when one retriever is confidently wrong, fusion can drag the wrong answer up. So hybrid is a strong default, not a guarantee, and on four test queries this is an illustration, not a benchmark. Test with your own questions.

Retrieval extras that improve quality

• Metadata filtering: restrict by user, date or document type before searching.
• MMR (maximal marginal relevance): pick results that are relevant but not duplicates of each other.
• Reranking: a stronger model (a cross-encoder) re-scores the top 20 to 50 results and keeps the best few. It costs a little delay and improves accuracy noticeably.
• Query rewriting: the LLM turns a vague chat message ("what about its price?") into a clear search query.
• Multi-query and query decomposition: search with several versions or sub-questions and merge the results.
• HyDE: the model writes a fake answer first and you search with that, since answers look more like documents than questions do.

Generate: the grounded answer

Now put it together. Retrieve the top 2 chunks, put them in the prompt, and instruct the model to use only them. The model here is a small 1.5-billion-parameter model running on this laptop's CPU (Session 23 shows how), so we can compare answers with and without retrieval:

import faiss
from llama_cpp import Llama
from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("all-MiniLM-L6-v2")
index = faiss.IndexFlatIP(384)
index.add(embedder.encode(DOCS, normalize_embeddings=True).astype("float32"))
llm = Llama(model_path="data/qwen2.5-1.5b-instruct-q4_k_m.gguf", n_ctx=2048, n_threads=8, verbose=False)

def ask(question, system):
    out = llm.create_chat_completion(messages=[{"role": "system", "content": system}, {"role": "user", "content": question}],
                                     max_tokens=80, temperature=0)
    return out["choices"][0]["message"]["content"].strip()

def rag(question, k=2):
    q = embedder.encode([question], normalize_embeddings=True).astype("float32")
    _, ids = index.search(q, k)
    context = "\n".join(f"- {DOCS[i]}" for i in ids[0])
    system = ("Answer using ONLY the context below. If the answer is not in the context, say you do not know.\n\nContext:\n" + context)
    return ask(question, system), [DOCS[i][:40] for i in ids[0]]

for question in ["What is the monthly fee for the pre-medical course?", "Can I get a refund after two weeks?"]:
    print("Q:", question)
    print("   without retrieval:", ask(question, "You are a helpful assistant for a tuition centre."))
    answer, used = rag(question)
    print("   with retrieval   :", answer)
    print("   chunks used      :", used, "\n")

Q: What is the monthly fee for the pre-medical course?
   without retrieval: I'm sorry, but I don't have specific information about the monthly fee for the pre-medical course at a particular tuition centre. The cost can vary depending on the institution, the location, and the specific program offered. To get accurate and up-to-date information, I recommend contacting the tuition centre directly or visiting their website. They should be able to provide you with the most current and relevant pricing
   with retrieval   : The monthly fee for the pre-medical course is Rs. 9,500.
   chunks used      : ['Monthly fee: the pre-medical course cost', 'Scholarship: students who score above 85'] 

Q: Can I get a refund after two weeks?
   without retrieval: I'm sorry, but as an AI language model, I don't have access to your specific tuition center's refund policy. Typically, refunds are only granted if there are extenuating circumstances such as illness, death in the family, or other unforeseen events. It's best to contact your tuition center directly to inquire about their refund policy. They can provide you with the most accurate and up-to
   with retrieval   : No, you cannot get a refund after two weeks.
   chunks used      : ['Refund policy: fees are refundable withi', 'Mock tests are held on the first Saturda']

The difference is the whole point of RAG. Asked the pre-medical fee without retrieval, the model politely says it does not know and suggests contacting the centre. With retrieval it answers "Rs. 9,500", straight from the document. For the refund question, retrieval gives "No, you cannot get a refund after two weeks", which matches the 7-day policy. Notice two further things. The retrieval step returned a second, irrelevant chunk each time (scholarship, mock tests), and a good model must ignore it. And the answer to the refund question is correct but terse: RAG controls what the model knows, and the answer style is still up to the prompt.

The rule "answer ONLY from the context; if it is not there, say you do not know" is called grounding, and it is the main guard against hallucination. It works well but is not perfect, so evaluate it.

import faiss
from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("all-MiniLM-L6-v2")
vecs = embedder.encode(chunks, normalize_embeddings=True).astype("float32")
index = faiss.IndexFlatIP(vecs.shape[1])          # inner product on unit vectors = cosine
index.add(vecs)

def rag(question, k=3):
    q = embedder.encode([question], normalize_embeddings=True).astype("float32")
    _, ids = index.search(q, k)
    context = "\n".join(chunks[i] for i in ids[0])
    return ask(question, system="Answer only from this context:\n" + context)

The same with a vector database that embeds, stores and searches for you:

import chromadb

col = chromadb.Client().create_collection("docs")
col.add(documents=chunks, ids=[str(i) for i in range(len(chunks))])
res = col.query(query_texts=["refund policy"], n_results=3)        # top-k chunks

Types of RAG

RAG has grown from a simple recipe into a family of designs:

A simple way to remember the progression: naive, then advanced (better retrieval), then corrective and self (checks quality), then agentic (plans and uses tools), with graph and multimodal changing what is retrieved.

Evaluating RAG

Test retrieval and generation separately, because a wrong answer can come from either:

• Context precision: are the retrieved chunks actually relevant?
• Context recall: did we retrieve everything needed?
• Faithfulness (groundedness): does the answer stay within the retrieved context?
• Answer relevance: does it address the question?

Start with a small set of real questions and expected answers. Tools such as RAGAS, TruLens, DeepEval and LangSmith help automate this.

Try it yourself

1. In chunking.py, change the chunk size to 80 and 300 characters. At what size do the chunks stop being useful?
2. Add three documents of your own to DOCS and ask questions about them in dense_search.py.
3. In hybrid_search.py, change the RRF constant k from 60 to 5. Does the hybrid score change on the four queries?
4. In rag_answer.py, ask a question the documents cannot answer ("What is the principal's name?"). Does the grounded model say it does not know?

What comes next

RAG lets a model read. Session 23 shows how to run models like the one we just used on an ordinary laptop, with no GPU. After that, Session 24 lets the model act.

Frequently asked questions