Local RAG Chatbot with Ollama

This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) chatbot that runs entirely locally. It loads a directory of Markdown (.md) files, creates a searchable vector index using local embeddings, and uses a local LLM (via Ollama) to answer questions based on your data.

Modules Used:

langchain: The framework for building LLM applications.
langchain-community: Community integrations (includes Ollama support).
langchain-chroma: Integration for the Chroma vector database.
ollama: The local LLM runner.

Prerequisites

Install Ollama: Download and install Ollama from ollama.com.
Pull Models: Open your terminal and pull the LLM and the embedding model:
```
ollama pull llama3
ollama pull nomic-embed-text
```
Data: A folder containing .md files (e.g., docs/ or notes/).

Installation

pip install langchain langchain-community langchain-chroma

The Code

Save this as local_doc_bot.py.

import argparse
import sys
import os

# LangChain Imports
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.chat_models import ChatOllama
from langchain_chroma import Chroma
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate

def create_knowledge_base(directory):
    print(f"Loading .md files from '{directory}'...")

    if not os.path.exists(directory):
        print(f"Error: Directory '{directory}' not found.")
        return None

    # Load documents
    loader = DirectoryLoader(directory, glob="**/*.md", loader_cls=TextLoader)
    docs = loader.load()

    if not docs:
        print("No markdown files found.")
        return None

    print(f"Loaded {len(docs)} documents. Splitting text...")

    # Split documents into chunks for embedding
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
    splits = text_splitter.split_documents(docs)

    print(f"Created {len(splits)} chunks. Building vector store (this may take a moment)...")

    # Create Vector Store using local embeddings (nomic-embed-text is efficient)
    vectorstore = Chroma.from_documents(documents=splits, embedding=OllamaEmbeddings(model="nomic-embed-text"))
    return vectorstore

def start_chat(vectorstore):
    # Initialize Local LLM
    llm = ChatOllama(model="llama3")

    # Create Retrieval Chain
    prompt = ChatPromptTemplate.from_template("""
    Answer the following question based only on the provided context:

    <context>
    {context}
    </context>

    Question: {input}
    """)

    document_chain = create_stuff_documents_chain(llm, prompt)
    retriever = vectorstore.as_retriever()
    retrieval_chain = create_retrieval_chain(retriever, document_chain)

    print("\n--- Local Doc Bot Ready (Type 'quit' to exit) ---")
    while True:
        query = input("\nQuestion: ")
        if query.lower() in ["quit", "exit"]:
            break

        print("Thinking...", end="", flush=True)
        response = retrieval_chain.invoke({"input": query})
        # Clear "Thinking..." line
        print("\r" + " " * 20 + "\r", end="")
        print(f"Answer: {response['answer']}")

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Local RAG Chatbot with Ollama")
    parser.add_argument("directory", help="Directory containing .md files")

    args = parser.parse_args()

    vs = create_knowledge_base(args.directory)
    if vs:
        start_chat(vs)

Usage

Prepare Data: Ensure you have a folder (e.g., my_docs) with some Markdown files inside.
Run the Bot:
```
python local_doc_bot.py ./my_docs
```
Ask Questions: The bot will answer based only on the information found in your Markdown files, running completely offline.

programming/python/python

Created: March 17, 2026 • Last Modified: January 28, 2026

Docs Login

Local RAG Chatbot with Ollama

Prerequisites

Installation

The Code

Usage