Embeddings & RAG
An LLM knows what was in its training data. It does not know your price list, your leave policy or yesterday's dispatch note. Retrieval-Augmented Generation fixes that by searching your documents first and passing the best passages to the model along with the question.
Embeddings: meaning as coordinates
An embedding model maps a piece of text to a fixed-length vector so that similar meanings land close together. "parcel not delivered" and "courier has not arrived yet" end up near each other even though they share no words. Closeness is measured by cosine similarity:
import numpy as np
def cosine(a, b):
a, b = np.asarray(a, dtype=float), np.asarray(b, dtype=float)
return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
query = [0.21, -0.04, 0.55]
chunks = {"refund policy": [0.20, -0.02, 0.58], "diwali offers": [-0.30, 0.44, 0.02]}
ranked = sorted(chunks, key=lambda k: cosine(query, chunks[k]), reverse=True)
print(ranked[0], round(cosine(query, chunks[ranked[0]]), 3))
Real vectors have hundreds or thousands of dimensions, but the arithmetic is exactly this.
The RAG pipeline
- Split documents into chunks (roughly 500–1,000 characters, with 10–15% overlap).
- Embed every chunk and store the vector plus the text and its metadata.
- Index in a vector store: FAISS, Chroma, pgvector inside PostgreSQL, or a managed service.
- Retrieve the top-k chunks for the user's question at query time.
- Generate an answer from those chunks, with citations.
chunks = split(docs, size=800, overlap=100)
index.add(embed(chunks))
context = index.search(embed(question), k=5)
prompt = f'''Answer using ONLY the context below. If the answer is not in it, say you do not know.
Cite the source id in square brackets after each claim.
Context:
{context}
Question: {question}'''
answer = llm(prompt)
Choosing a store
| Option | Good for | Watch out for |
|---|---|---|
| FAISS | Fast local prototypes, read-heavy | No metadata filtering or persistence out of the box |
| Chroma | Small apps, easy start | Operational limits as the corpus grows |
| pgvector | Teams already running PostgreSQL | Index tuning matters past a few hundred thousand rows |
| Managed vector DB | Scale, hybrid search, filters | Cost, and where the data physically sits |
Making retrieval actually good
Most RAG failures are retrieval failures, not model failures. Improve them in this order:
- Chunk on structure — split on headings and clauses, not blindly on character counts. Keep a table with its caption.
- Store metadata and filter on it:
branch = "Surat",effective_from <= today,language = "hi". - Hybrid search — combine keyword (BM25) with vector search so exact terms like an SKU or a GSTIN still match.
- Rerank the top 30 hits with a cross-encoder and keep the best 5.
- Test retrieval separately: for 30 real questions, is the correct passage in the top 5? That is recall@5, and if it is 60% no prompt will save you.
Common mistakes
- Embedding the question with one model and the chunks with another; the vectors are not comparable.
- Chunks so large that five of them exhaust the context window, or so small that the answer is cut in half.
- Never re-indexing after a document is updated, so the bot quotes last year's prices.
- Forgetting access control: retrieval must respect who is allowed to see which document.
- Skipping the "say you do not know" instruction, which invites confident invention.
Practice
Take 20 pages of any policy PDF you have. Chunk it two ways — fixed 800 characters, and split on headings — and write 15 questions with the passage that should answer each. Measure recall@5 for both chunkings and keep the winner.