Part 1 covered models and messages. Part 2 covered prompt templates and output parsers. Both had the same limitation: the model only knows what was in its training data. Ask llama3.2:3b about your BGP peering runbook and it will produce a fluent, confident, entirely invented answer.
Retrieval augmented generation, almost always shortened to RAG, is the fix. Before you call the model, you search your own documents for the passages most relevant to the question, and you paste those passages into the prompt. The model then answers from text you supplied rather than from memory.
This part builds that pipeline piece by piece and ends with a working chain. Everything still runs locally with Ollama.
The Retrieval Pipeline
RAG has five stages, and each one maps to a LangChain abstraction:
- Load. Turn files, pages, or database rows into
Documentobjects. Document loaders do this. - Split. Cut documents into chunks small enough to embed and specific enough to match a question. Text splitters do this.
- Embed. Convert each chunk into a vector of numbers that encodes its meaning. Embedding models do this.
- Store. Keep the vectors somewhere you can search by similarity. Vector stores do this.
- Retrieve. Given a question, return the closest chunks. Retrievers do this.
Stages one through four run once, when you ingest your data. Stage five runs on every question. Keep that split in mind, because it is the difference between a script that takes ten minutes and a query that takes two seconds.
Install what you need:
pip install langchain langchain-ollama langchain-text-splitters langchain-community beautifulsoup4A note on
langchain-community: this package was archived in June 2026 and is no longer maintained. It still installs and works, and it remains the shortest path to the loaders below. New integrations now ship as standalone packages instead. Pin the version if you depend on it, and read the section on hand-written loaders before you build anything long-lived on top of it.
The examples use two small runbook files in a runbooks/ directory. Any plain text you have will work.
Document Loaders
Everything in the retrieval pipeline operates on Document objects. A Document has exactly two attributes that matter: page_content, which is the text, and metadata, which is a dict describing where the text came from.
TextLoader reads a single file:
from langchain_community.document_loaders import TextLoader
docs = TextLoader("runbooks/bgp.txt").load()
print("documents:", len(docs))
print("metadata:", docs[0].metadata)
print(repr(docs[0].page_content[:120]))documents: 1
metadata: {'source': 'runbooks/bgp.txt'}
'BGP peering runbook.\n\nSite DFW1 peers with two upstream transit providers over eBGP. The local ASN is 64512.\nHold timer 'The source key is set automatically. That metadata is how you cite an answer later, so treat it as important rather than incidental.
DirectoryLoader walks a tree and applies a loader to each match:
from langchain_community.document_loaders import DirectoryLoader, TextLoader
docs = DirectoryLoader("runbooks", glob="**/*.txt", loader_cls=TextLoader).load()
print("documents:", len(docs))
print("sources:", sorted(d.metadata["source"] for d in docs))documents: 2
sources: ['runbooks/bgp.txt', 'runbooks/ospf.txt']Other loaders worth knowing: PyPDFLoader reads a PDF and returns one document per page, which requires pip install pypdf. CSVLoader returns one document per row, which suits ticket exports and inventory dumps.
Writing your own loader
Given that langchain-community is frozen, it is worth knowing that a loader is not magic. A Document is a plain object, and building a list of them yourself is a few lines:
from pathlib import Path
from langchain_core.documents import Document
docs = [
Document(page_content=path.read_text(), metadata={"source": str(path)})
for path in Path("runbooks").glob("*.txt")
]Everything downstream works identically. For simple sources, this is often the better answer.
Web Loaders
WebBaseLoader fetches a URL and extracts its text. It needs beautifulsoup4.
Caution: Scraping a site you do not own has rules. Check
robots.txt, respect the terms of service, set a descriptiveUSER_AGENTenvironment variable, and rate limit yourself. Do not point a recursive loader at somebody else’s site without permission.
Without filtering you get the whole page, navigation and footer included. The bs_kwargs argument narrows the extraction to the element that holds the real content:
import bs4
from langchain_community.document_loaders import WebBaseLoader
loader = WebBaseLoader(
web_paths=["https://jtdub.com/2025/01/24/ai-transformers/"],
bs_kwargs={"parse_only": bs4.SoupStrainer("article")},
)
docs = loader.load()
print("documents:", len(docs))
print("metadata:", docs[0].metadata)
print("characters:", len(docs[0].page_content))
print(repr(docs[0].page_content.strip()[:160]))documents: 1
metadata: {'source': 'https://jtdub.com/2025/01/24/ai-transformers/'}
characters: 6980
'Engineering & Code What Are Tokenizers in AI, and Why Are They Important? January 24, 2025 5 min read Python · Open Source · Python Tips Artificial intel'That SoupStrainer("article") call is doing real work. Skip it and you index the site navigation on every page, which pollutes every search that follows.
RecursiveUrlLoader follows links from a starting URL down to a max_depth. It is useful for ingesting a documentation site, and it is also the fastest way to accidentally hammer a server. Set max_depth low and start with a narrow path.
Splitting Documents
Feeding a whole document into an embedding model is wrong for two reasons. The model has an input limit, so long documents get truncated. More importantly, a single vector averaged over 7,000 characters represents everything and matches nothing. Retrieval quality collapses.
RecursiveCharacterTextSplitter is the default choice. It tries to split on paragraph breaks first, then line breaks, then sentences, then words, so chunks tend to land on natural boundaries:
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
docs = DirectoryLoader("runbooks", glob="**/*.txt", loader_cls=TextLoader).load()
splitter = RecursiveCharacterTextSplitter(chunk_size=400, chunk_overlap=60)
chunks = splitter.split_documents(docs)
print("chunks:", len(chunks))
for chunk in chunks[:2]:
print(" -", len(chunk.page_content), "chars |", repr(chunk.page_content[:70]))chunks: 3
- 376 chars | 'OSPF runbook.\n\nThe campus runs a single OSPF process with area 0 in th'
- 212 chars | 'BGP peering runbook.\n\nSite DFW1 peers with two upstream transit provid'Two parameters control the result:
chunk_sizeis the maximum characters per chunk. Small chunks retrieve precisely but lose context. Large chunks carry context but dilute the match. Start at 1000 for prose and 500 for reference material like runbooks or API docs.chunk_overlaprepeats the tail of one chunk at the head of the next, so a sentence that spans a boundary still appears intact somewhere. Set it to roughly 10 to 20 percent ofchunk_size.
split_documents() copies the metadata onto every chunk, so each one still knows its source file.
Embedding Models
An embedding is a list of floating point numbers that encodes the meaning of a piece of text. Texts about similar things end up close together in that number space, which is what makes similarity search work. “How do I clear a BGP session” and “resetting a peer” land near each other even though they share no words.
OllamaEmbeddings runs an embedding model locally:
from langchain_ollama import OllamaEmbeddings
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vector = embeddings.embed_query("How do I clear a BGP session?")
print("dimensions:", len(vector))
print("first 5:", [round(x, 4) for x in vector[:5]])
vectors = embeddings.embed_documents(["OSPF stub area", "BGP prefix limit"])
print("embed_documents ->", len(vectors), "vectors of", len(vectors[0]))dimensions: 768
first 5: [0.0407, 0.0435, -0.1445, -0.0221, 0.0529]
embed_documents -> 2 vectors of 768There are two methods because some models embed questions and passages differently. Use embed_query() for the user’s question and embed_documents() for your corpus. LangChain calls the right one for you inside a vector store.
Two rules matter more than anything else here.
Use the same model for both sides. A vector from nomic-embed-text is meaningless to a store built with a different model. Change embedding models and you must re-embed the entire corpus. Record which model you used alongside the data.
Use a dedicated embedding model. A chat model can produce embeddings, but a purpose-built one is smaller, faster, and better at the job. nomic-embed-text is 274 MB and produces 768 dimensions. Running llama3.2:3b for the same task would be seven times larger and worse.
Vector Stores
A vector store holds vectors alongside their text and metadata, and searches them by similarity. InMemoryVectorStore ships with langchain-core and requires no setup, which makes it ideal for learning:
from langchain_core.vectorstores import InMemoryVectorStore
store = InMemoryVectorStore.from_documents(chunks, embeddings)
hits = store.similarity_search("What is the inbound prefix limit?", k=2)
for hit in hits:
print(" *", hit.metadata["source"], "|", repr(hit.page_content[:90])) * runbooks/bgp.txt | 'If a session flaps, first check the physical interface for errors, then check the\nprefix l'
* runbooks/ospf.txt | 'OSPF runbook.\n\nThe campus runs a single OSPF process with area 0 in the core and stub area'from_documents() embeds every chunk and indexes it in one call. similarity_search() embeds the query, compares it against everything stored, and returns the k closest documents.
When you need to know how good a match is, ask for scores:
for doc, score in store.similarity_search_with_score("adjacency stuck in EXSTART", k=2):
print(f" score={score:.4f} {doc.metadata['source']} {repr(doc.page_content[:60])}") score=0.4442 runbooks/ospf.txt 'OSPF runbook.\n\nThe campus runs a single OSPF process with ar'
score=0.4180 runbooks/bgp.txt 'BGP peering runbook.\n\nSite DFW1 peers with two upstream tran'Scores let you set a floor and return nothing rather than returning the least-bad match. Note that the scale differs between stores, so calibrate against your own data instead of copying a threshold from a tutorial.
InMemoryVectorStore disappears when your process exits. For anything real, pick a store that persists:
| Store | Install | Use it when |
|---|---|---|
InMemoryVectorStore | included in langchain-core | Learning, tests, small one-shot scripts |
| Chroma | pip install langchain-chroma | A local app that needs persistence with almost no setup |
| FAISS | pip install faiss-cpu | Large local corpora where search speed matters |
| pgvector, Qdrant, Pinecone, Weaviate | separate packages | Shared access, filtering at scale, or a corpus too large for one machine |
The interface is nearly identical across all of them, so starting in memory and switching later is a small change.
Retrievers
A retriever is the interface a chain actually consumes. It takes a string and returns a list of documents, and nothing else. Any vector store gives you one:
retriever = store.as_retriever(search_type="similarity", search_kwargs={"k": 2})
print([d.metadata["source"] for d in retriever.invoke("how do I clear a bgp session safely")])['runbooks/bgp.txt', 'runbooks/bgp.txt']Both hits came from the BGP runbook, which is correct for that question. search_kwargs={"k": 2} sets how many documents come back.
Three search types are available:
similarityis the default. It returns theknearest chunks.similarity_score_thresholdadds ascore_thresholdand drops anything below it, so a question with no good answer returns an empty list.mmrstands for maximal marginal relevance. It fetchesfetch_kcandidates, then pickskof them that are relevant and different from each other.
MMR matters when your corpus contains near-duplicates. Plain similarity search will happily return four chunks that all say the same thing, filling the context window with one fact. MMR trades a little relevance for coverage:
mmr = store.as_retriever(search_type="mmr", search_kwargs={"k": 2, "fetch_k": 6})
print([d.metadata["source"] for d in mmr.invoke("routing timers and areas")])['runbooks/bgp.txt', 'runbooks/ospf.txt']One from each runbook, which is what a question spanning both topics should return.
Two retrievers worth knowing once the basics work. MultiQueryRetriever asks a model to rewrite the question several ways, runs all of them, and merges the results, which helps when users phrase things differently from your documents. ContextualCompressionRetriever runs retrieved chunks through a second pass that strips the irrelevant parts before they reach the prompt. Both live in langchain_classic.retrievers as of version 1.0.
A Complete RAG Chain
Everything assembled. This loads a directory, splits it, embeds it, stores it, and answers questions from the retrieved text:
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
docs = DirectoryLoader("runbooks", glob="**/*.txt", loader_cls=TextLoader).load()
chunks = RecursiveCharacterTextSplitter(
chunk_size=400, chunk_overlap=60).split_documents(docs)
store = InMemoryVectorStore.from_documents(
chunks, OllamaEmbeddings(model="nomic-embed-text"))
retriever = store.as_retriever(search_kwargs={"k": 3})
prompt = ChatPromptTemplate.from_messages([
("system",
"Answer the question using only the context below. "
"If the context does not contain the answer, say you do not know.\n\n"
"Context:\n{context}"),
("human", "{question}"),
])
def format_docs(docs):
return "\n\n".join(d.page_content for d in docs)
chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| ChatOllama(model="llama3.2:3b", temperature=0)
| StrOutputParser()
)
for question in [
"What is the inbound prefix limit on a transit session?",
"What causes an OSPF adjacency to stay in EXSTART?",
"What is the WiFi password?",
]:
print(f"Q: {question}\nA: {chain.invoke(question)}\n")Q: What is the inbound prefix limit on a transit session?
A: The inbound prefix limit on a transit session is 900000.
Q: What causes an OSPF adjacency to stay in EXSTART?
A: An OSPF adjacency that stays in EXSTART almost always means an MTU mismatch between the two ends.
Q: What is the WiFi password?
A: I don't know. The provided context only discusses network protocols and runbooks for BGP and OSPF, but does not mention WiFi or passwords.The dict at the head of the chain is the piece worth studying. LCEL treats a dict of runnables as a parallel step: it runs both branches on the same input and produces a dict of results. The context branch sends the question to the retriever and formats the documents into a string. The question branch is RunnablePassthrough(), which forwards the input unchanged. The prompt template then receives exactly the two keys it declared.
The third answer is the one that proves the system works. The model refused to invent a WiFi password because the system message told it to answer only from context, and the context contained nothing relevant. A model without retrieval would have made something up.
Where This Falls Short
A 3B model has real limits and you will meet them quickly. It follows the “answer only from context” instruction most of the time, not all of the time. It handles two or three retrieved chunks well and gets confused by ten. Its context window is small enough that a k of 8 with 1000-character chunks starts crowding out the question.
None of that is a reason to avoid running locally. It is a reason to be deliberate: retrieve fewer, better chunks rather than more, keep chunks tight, and write the system message defensively. If quality is still short, the same chain runs against a larger model by changing one line, because that was the point of the abstraction in the first place.
From here the useful next steps are adding source citations by reading metadata off the retrieved documents, persisting the store with Chroma so ingestion runs once instead of on every start, evaluating retrieval quality against a fixed set of questions, and giving the model tools so it can decide when to search rather than always searching. That last one turns a chain into an agent, which is where the create_agent function in the top-level langchain package comes in.
The three parts of this series cover the pieces most LangChain applications are built from. A model wrapper, typed messages, templated prompts, parsers that produce real objects, and a retrieval pipeline that grounds the model in your own data. Everything ran on a laptop with no API key, which makes it cheap to experiment with until you know what you actually need.