How to Build a RAG System with LangChain and Python

Ask an LLM about your company's vacation policy and it will confidently invent one. The model has never seen your documents: its knowledge froze at training time, and it fills the gaps with plausible fiction.

Retrieval-Augmented Generation (RAG) fixes this by searching your own documents for the fragments relevant to each question and injecting them into the prompt, so the model answers from evidence instead of memory. In this guide you'll build a complete RAG system with LangChain and Python — and, more importantly, cover the parts most tutorials skip: how to choose a chunking strategy, how to build an evaluation dataset, and the failure modes that show up in production.

How a RAG System Works

The architecture has two phases:

1. Indexing (offline): split documents into chunks, convert each chunk into an embedding — a vector that captures its meaning — and store the vectors in a vector database.
2. Retrieval and generation (per question): embed the question, find the most similar chunks, and pass them to the LLM as context alongside the question.

Everything that follows maps to one of these two phases — and so does almost every quality problem: bad answers usually trace back to bad retrieval, and bad retrieval usually traces back to bad chunking.

Installation

pip install langchain langchain-community langchain-openai langchain-chroma tiktoken

Indexing Documents into a Vector Store

from langchain_community.document_loaders import DirectoryLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

# Load documents
loader = DirectoryLoader("./docs", glob="**/*.pdf")
documents = loader.load()

# Split into 1000-character chunks with 200-character overlap
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
)
chunks = splitter.split_documents(documents)

# Create the vector store
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory="./chroma_db")

Two details matter here. First, chunk_size is measured in characters, not tokens — if you want token-based sizes, use RecursiveCharacterTextSplitter.from_tiktoken_encoder(). Second, the overlap (chunk_overlap) ensures that an idea cut at a chunk boundary survives complete in at least one of the two neighboring chunks.

Chunking Strategies: Where Retrieval Quality Is Decided

chunk_size=1000 is a reasonable default, not a law. Chunking is the single decision with the most impact on retrieval quality, so it's worth understanding the trade-off:

• Small chunks (200–400 tokens) produce precise embeddings — each vector represents one idea — but often lack the context the LLM needs to answer. Retrieval succeeds, generation fails.
• Large chunks (1,500+ tokens) carry plenty of context but dilute the embedding: a chunk that mixes three topics matches queries about any of them poorly.
• An overlap of 10–20% of the chunk size is enough insurance against boundary cuts; much more than that mostly duplicates your index and your embedding bill.

Just as important: match the splitter to the document structure.

1. Prose (policies, manuals, articles): RecursiveCharacterTextSplitter works well because it tries to split on paragraphs before sentences, and sentences before words.
2. Markdown or HTML with headings: split by heading first (MarkdownHeaderTextSplitter) and attach the heading path as metadata. A chunk that only says "up to 5 days may be carried over" is useless unless it knows it lives under "Vacations → Carryover".
3. Tables: never let a splitter cut rows in half. Extract tables separately and convert each row (or small group of rows) into a sentence before embedding.
4. Code: split by function or class with a language-aware splitter, not by character count.

My practical rule: don't argue about chunk sizes in the abstract. Index the same corpus with two or three configurations, run your evaluation questions (below) against each one, and let the metrics decide.

Building the Retrieval Pipeline

from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQAWithSourcesChain

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

retriever = vectorstore.as_retriever(
    search_type="mmr",  # Maximum Marginal Relevance for diversity
    search_kwargs={"k": 5, "fetch_k": 20},
)

chain = RetrievalQAWithSourcesChain.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True,
)

We use MMR (Maximum Marginal Relevance) instead of pure cosine similarity: it fetches fetch_k=20 candidates, then selects k=5 to maximize relevance and diversity, which avoids retrieving five near-identical chunks from the same document.

Querying the System

result = chain.invoke({"question": "What is the vacation policy?"})

print(result["answer"])
print("Sources:", result["sources"])

Returning sources isn't cosmetic: it lets users verify answers, and it lets you debug retrieval when an answer goes wrong.

Evaluation: The Step Everyone Skips

A RAG system without evaluation is a system you don't know works. Use RAGAS to measure, at minimum:

• Faithfulness: Is the answer supported by the retrieved documents?
• Answer Relevancy: Does the answer actually address the question?
• Context Recall: Were the right chunks retrieved?

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall

results = evaluate(
    dataset=test_dataset,
    metrics=[faithfulness, answer_relevancy, context_recall],
)

How to Build the Evaluation Dataset

RAGAS needs a test set of questions, ideally with reference answers. Three sources, in order of value:

1. Real user questions. If the system replaces an FAQ or a support flow, mine 30–50 questions from tickets or chat logs. They beat anything you invent because they come with real phrasing, typos, and ambiguity.
2. Synthetic questions for coverage. Use an LLM to generate question–answer pairs from random chunks of the corpus (RAGAS ships a TestsetGenerator for this). Review them by hand: synthetic questions tend to be suspiciously well aligned with the text that produced them, which inflates your metrics.
3. Unanswerable questions. Make 10–15% of the set questions your corpus cannot answer. The correct behavior is "I don't know" — measure whether you actually get it.

Between 50 and 100 curated questions is enough to catch regressions. Version this dataset next to your code and re-run it after every change to chunking, embeddings, or prompts. Without that habit, every "improvement" is a blind redeploy.

Common RAG Failure Modes in Production

These are the problems I see most often once a prototype meets real users, along with the standard fixes:

1. Stale index. Documents change; the index doesn't. Schedule re-indexing triggered by the document source, and store each document's last-modified date as metadata so you can audit freshness.
2. Lost in the middle. Cranking k up to 20 chunks makes answers worse, not better: models pay less attention to the middle of a long context. Retrieve fewer, better chunks — and add a re-ranking step (a cross-encoder reordering the retriever's candidates) if precision is the bottleneck.
3. Answering the unanswerable. Vector search always returns something — the nearest neighbor exists even when it's irrelevant. Set a similarity threshold, and explicitly instruct the model to say it doesn't know when the context doesn't support an answer.
4. No metadata filtering. "What's the 2025 policy?" happily retrieves the 2019 version, because both are semantically identical. Filter by metadata (year, department, document type) before the similarity search.
5. Cost surprises. Embedding is cheap; answer generation isn't. Semantic caching (reusing answers to similar questions) and routing easy questions to a smaller model cut the bill significantly.

When NOT to Use RAG

RAG isn't the answer to everything. Consider alternatives if:

• You have a few dozen short documents: with modern context windows you can often pass everything directly in the prompt — no vector search needed.
• Knowledge is extremely dynamic (real-time data): use function calling + APIs instead of constantly re-indexing.
• Questions require multi-step reasoning or computation: an agent with tools works better — I show how to build one in AI agents with LangGraph for data analysis.
• You need to extract specific fields from documents rather than answer open questions: that's structured data extraction with LLMs, a simpler and more reliable pattern.

Conclusion

RAG remains the most pragmatic way to connect an LLM to private knowledge, and LangChain gets you a working prototype in an afternoon. What separates a demo from a production system is everything around the pipeline: chunking chosen for your documents' structure, an evaluation dataset built from real questions, and metrics you re-run before every change. Invest there first — retrieval quality, not prompt wording, is where most RAG systems win or lose.

Are you implementing RAG at your company? Let's talk — I can help you avoid the most common mistakes.