How to Build A RAG System Using Claude: An AI That Runs on Your Own Data (Full Guide)

Ask Claude about your company, your notes, or your files, and it draws a blank. It never saw them. It only knows what it learned in training, and your stuff wasn't part of that.

A RAG system fixes this. Instead of answering from memory, Claude first looks things up in your documents, grabs the parts that matter, and answers from what it actually found. Your data, its source, no guessing.

Why it beats just pasting files into the chat:

It scales. Your whole knowledge base won't fit in one chat. RAG stores everything and pulls only what each question needs.

It's cheaper. Pasting a file means Claude re-reads the entire thing on every single question. RAG reads it once, then fetches only the relevant part. Instead of sending a 10,000-token manual every time, it might send 500 tokens of the exact section you need. Real setups cut token use by 80% or more.

It's sharper. Feed a model a giant wall of text and it loses details in the middle. Hand it a few precise chunks and answers get more accurate.

It stays current. Update your files once and the system uses the fresh version. No re-pasting.

By the end of this guide, you'll have one running on your own files, step by step, no PhD required.

What you'll need

Before we touch any code, here's the full list. Good news: this version needs just one API key, and everything else runs free on your own machine.

1. Python 3.9 or newer. To check if you have it, open your terminal (Terminal on Mac, Command Prompt on Windows) and type python --version. If you see something like 3.11, you're good. If not, download it from python.org and run the installer. On Windows, tick the box "Add Python to PATH" during setup, otherwise the commands below won't work.

2. One Claude API key, plus a small credit balance. This is the only key and the only money the whole guide needs. Here's the exact path, click by click:

Go to platform.claude.com, and log in (or sign up) there.

The API needs a positive balance to run, so add funds first. When prompted, pick whether the credits are for yourself or a company, then you land on the payment screen. Choose the $5 "Starting out" option. That's plenty: everything else in this guide is free and local, so Claude is the only thing that costs money, and each question runs you a fraction of a cent. Credits expire one year after purchase.

After paying, you'll land on your Console dashboard. You should see your balance (for example $5.00) in the top left under "Organization credits."

Now grab the key. Click Get API key (top right), then Create Key. Give it any name you like (for example my-rag-key) and leave the workspace as Default. Click create, then copy the string it shows you. It starts with sk-ant- and you only see it once, so paste it somewhere safe for a minute.

That's the entire setup.

Step 1: Add your key and load your files

1. Make the project folder. Create a new folder on your Desktop and name it rag-project. Everything goes here.

2. Open your terminal. On Mac: Cmd+Space, type Terminal, enter. On Windows: Start button, type cmd, enter.

3. Point the terminal at your folder. Type cd and a space, then drag the rag-project folder onto the terminal window and press enter. Every command below is run from inside this folder.

cd Desktop/rag-project

4. Install the tools. Paste this into your terminal and press enter (first run may take a minute):

pip install anthropic chromadb sentence-transformers pypdf python-dotenv

If you get pip: command not found, use pip3 instead of pip. When the terminal shows a fresh line with no red errors, it's done.

5. Create your code file. Inside rag-project, create an empty file named exactly rag.py. Open it in any text editor.

6. Create your key file. In the same folder, create a file named exactly .env (starts with a dot, no name before it). Paste this inside, with the real key you made during setup after the =, no spaces, no quotes:

ANTHROPIC_API_KEY=sk-ant-paste-your-real-key-here

Keeping the key in .env instead of in your code means it won't leak if you share the script or put it on GitHub.

7. Load the key. Put this at the top of rag.py:

import os
from dotenv import load_dotenv

load_dotenv()  # reads your .env file
api_key = os.getenv("ANTHROPIC_API_KEY")

8. Create your knowledge base. Inside rag-project, create a folder named documents. Drop any .txt, .md, or .pdf files in there: your notes, a product doc, meeting summaries, anything.

8.1. If you don't have files yet use this test file. Create notes.txt inside the documents folder and paste this in:

Project Northstar is our internal tool for tracking customer feedback. It was launched in March 2026 and is maintained by the platform team. The lead engineer is Dana Reyes. Feedback is reviewed every Friday. Northstar replaced the old spreadsheet system we used through 2025.

At the end you'll ask Claude about Northstar and watch it answer from this exact file.

9. Add the code that reads your files. Below the code from step 7, in rag.py:

from pathlib import Path
from pypdf import PdfReader

def load_documents(folder="documents"):
    docs = []
    for file in Path(folder).iterdir():
        if file.suffix in [".txt", ".md"]:
            text = file.read_text(encoding="utf-8")
            docs.append({"source": file.name, "text": text})
        elif file.suffix == ".pdf":
            reader = PdfReader(str(file))
            text = "\n".join(page.extract_text() or "" for page in reader.pages)
            docs.append({"source": file.name, "text": text})
    return docs

documents = load_documents()
print(f"Loaded {len(documents)} document(s).")

10. Run it. Save rag.py, then in your terminal:

python rag.py

You should see:

Loaded 1 document(s)

If you see Loaded 0 document(s), the documents folder is empty or in the wrong place. It must sit directly inside rag-project, next to rag.py.

Step 2: Split your files into chunks

Right now each file is one big block of text. Before we can search it, we need to cut it into smaller pieces called chunks. Here's why: when someone asks a question, the system finds the chunks that match and sends only those to Claude. If your chunks are whole 50-page documents, you send way too much. If they're single sentences, they lose context. Small paragraphs are the sweet spot.

1. Add the chunking code. Below the code from step 10, in rag.py:

def chunk_text(text, chunk_size=500, overlap=100):
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunk = " ".join(words[start:end])
        chunks.append(chunk)
        start = end - overlap  # step back a little so chunks overlap
    return chunks

Two numbers to understand here, in plain terms:

2. Turn every document into chunks. Add this below:

all_chunks = []
for doc in documents:
    for chunk in chunk_text(doc["text"]):
        all_chunks.append({"source": doc["source"], "text": chunk})

print(f"Created {len(all_chunks)} chunk(s) from {len(documents)} document(s).")

Notice each chunk carries its source (the filename it came from). We keep that attached the whole way through, so when Claude answers later, it can tell you which file the answer came from.

3. Run it. Save rag.py, then in your terminal:

python rag.py

You should see something like:

Loaded 1 document(s).

Created 1 chunk(s) from 1 document(s).

The small test file becomes just one chunk because it's short. Real documents will produce many. If you dropped a long PDF in the folder, you might see dozens or hundreds of chunks, which is exactly what you want.

Step 3: Turn your chunks into embeddings

This is the step that lets the computer search by meaning instead of exact words. Each chunk gets converted into a list of numbers (an embedding) that captures what it's about. Chunks with similar meaning end up with similar numbers. Later, when a question comes in, we turn the question into numbers too and find the closest matches.

The model that does this runs locally on your machine. It downloads once, then works offline and free, and your files never leave your computer.

1. Load the embedding model. Below the code from step 2, in rag.py:

from sentence_transformers import SentenceTransformer

print("Loading the embedding model (first run downloads it, about 90 MB)...")
embedder = SentenceTransformer("all-MiniLM-L6-v2")

The very first time you run this, it downloads the model, so give it a moment. Every run after that is instant because it's already on your machine.

2. Turn every chunk into an embedding. Add this below:

chunk_texts = [chunk["text"] for chunk in all_chunks]
embeddings = embedder.encode(chunk_texts)

print(f"Created {len(embeddings)} embedding(s).")
print(f"Each embedding is a list of {len(embeddings[0])} numbers.")

embedder.encode(...) takes your list of chunk texts and hands back one embedding per chunk. That's all it takes.

3. Run it. Save rag.py, then in your terminal:

python rag.py

The first run pauses while the model downloads, then you should see something like:

Loaded 1 document(s).

Created 1 chunk(s) from 1 document(s).

Loading the embedding model (first run downloads it, about 90 MB)...

Created 1 embedding(s).

Each embedding is a list of 384 numbers.

That "384 numbers" line is the whole idea made visible: your text is now a row of numbers the computer can compare. You don't need to read or understand those numbers yourself. The database in the next step handles all the comparing for you.

If the download fails with a connection error, just run the command again. It picks up where it left off.

Step 4: Store everything in your vector database

Now we put the chunks and their embeddings into Chroma, your local database. This is what makes search fast: instead of comparing your question against every chunk by hand each time, Chroma stores them ready to go and does the matching for you. It saves to a folder on your machine, so you only build it once.

1. Set up the database. Below the code from step 3, in rag.py:

import chromadb

client = chromadb.PersistentClient(path="chroma_db")
collection = client.get_or_create_collection("my_documents")

PersistentClient(path="chroma_db") tells Chroma to save into a folder called chroma_db (it creates it automatically, right next to your script). Because it's saved to disk, your data survives after the script finishes. A collection is just the named box your chunks live in.

2. Add your chunks to the database. Add this below:

collection.add(
    ids=[str(i) for i in range(len(all_chunks))],
    embeddings=[emb.tolist() for emb in embeddings],
    documents=[chunk["text"] for chunk in all_chunks],
    metadatas=[{"source": chunk["source"]} for chunk in all_chunks],
)

print(f"Stored {collection.count()} chunk(s) in the database.")

Here's what each line hands to Chroma, in plain terms: ids gives every chunk a unique label (0, 1, 2...), embeddings is the numbers from step 3, documents is the actual chunk text, and metadatas carries the filename along so we can show the source later. Chroma keeps all four tied together.

3. Run it. Save rag.py, then in your terminal:

python rag.py

You should see:

Stored 1 chunk(s) in the database.

One thing to know for later. Every time you run the script right now, it adds the chunks again, so counts can climb (1, then 2, then 3...) on repeat runs. That's fine while we're building. To start clean, delete the chroma_db folder and run once more. In the final version we'll handle this properly so it doesn't double up.

Step 5: Search your documents

This is the "retrieval" part of RAG, the R in the name. We take a question, turn it into an embedding the same way we did the chunks, and ask Chroma for the chunks whose meaning is closest. Those matching chunks are what we'll hand to Claude in the next step.

1. Add the search function. Below the code from step 4, in rag.py:

def search(question, n_results=3):
    question_embedding = embedder.encode([question])[0]
    results = collection.query(
        query_embeddings=[question_embedding.tolist()],
        n_results=n_results,
    )
    return results

What this does, line by line in plain terms: it turns the question into numbers with the same model you used on your chunks (this is important, both have to speak the same "number language"), then asks Chroma for the closest matches. n_results=3 means "give me the 3 most relevant chunks." Three is a good default: enough context, not so much that you waste tokens.

2. Try a search. Add this below to test it:

question = "Who runs Northstar and when is feedback reviewed?"
results = search(question)

for i, doc in enumerate(results["documents"][0]):
    source = results["metadatas"][0][i]["source"]
    print(f"\n--- Match {i+1} (from {source}) ---")
    print(doc)

This runs a real question against your database and prints the chunks it found, each with the filename it came from.

3. Run it. Save rag.py, then in your terminal:

python rag.py

With the Northstar test file, you should see it pull back the matching chunk, something like:

--- Match 1 (from notes.txt) ---

Project Northstar is our internal tool for tracking customer feedback. It was launched in March 2026 and is maintained by the platform team. The lead engineer is Dana Reyes. Feedback is reviewed every Friday. Northstar replaced the old spreadsheet system we used through 2025.

Notice what just happened: your question used the words "who runs" and "reviewed," but the file says "lead engineer" and "reviewed every Friday." It matched anyway, because the search works on meaning, not exact words. That's the whole point of embeddings, and it's why this beats a plain keyword search (Ctrl+F) through your files.

If you have more files, you'll see the top 3 chunks from across all of them, sorted by how closely they match.

Step 6: Get Claude to answer from what it found

This is the "generation" part, the G in RAG. We take the chunks from step 5, hand them to Claude Opus 4.8 along with the question, and tell it to answer using only that context. This is what stops it from guessing: Claude answers from your files, not from its own memory, and tells you which file it used.

1. Add the answer function. Below the code from step 5, in rag.py:

import anthropic

claude = anthropic.Anthropic(api_key=api_key)

def answer(question):
    results = search(question)
    chunks = results["documents"][0]
    sources = [m["source"] for m in results["metadatas"][0]]

    context = ""
    for i, chunk in enumerate(chunks):
        context += f"[From {sources[i]}]\n{chunk}\n\n"

    message = claude.messages.create(
        model="claude-opus-4-8",
        max_tokens=1024,
        system=(
            "You answer questions using only the context provided. "
            "If the answer is not in the context, say you don't know. "
            "Always mention which file your answer came from."
        ),
        messages=[
            {
                "role": "user",
                "content": f"Context:\n{context}\nQuestion: {question}",
            }
        ],
    )
    return message.content[0].text

What's happening here, in plain terms: we search for the relevant chunks, glue them together into one context block (each labeled with its filename), then send that block plus the question to Claude. The system instruction is the key part. It tells Claude three things: answer only from the context, admit when the answer isn't there, and name the source file. Those three rules are what make the answers trustworthy instead of made up.

model="claude-opus-4-8" is the exact model name (dashes, not dots). max_tokens=1024 caps how long the answer can be.

2. Ask a question. Add this below:

question = "Who runs Northstar and when is feedback reviewed?"
print(answer(question))

3. Run it. Save rag.py, then in your terminal:

python rag.py

You should get a real answer built from your file, something like:

Dana Reyes is the lead engineer who runs Project Northstar, and feedback is reviewed every Friday. (Source: notes.txt)

That's a full RAG system working. Claude never saw this file during training, it can't know who Dana Reyes is, yet it answered correctly and told you exactly where the answer came from. Ask it something that isn't in your files and it will say it doesn't know, instead of inventing an answer. That "I don't know" is a feature, not a failure: it's the difference between a tool you can trust and one that guesses.

Step 7: Turn it into something you can actually use

Right now you have to edit the code and rerun the whole script every time you want to ask something. Worse, every run re-reads your files and re-adds them to the database, so chunks pile up. Let's fix both: build the database only once, then let you ask questions on a loop, typing them straight into the terminal.

1. Fix the double-adding. Find the block from step 4 that adds chunks (the collection.add(...) part) and replace it with this version, which only builds the database if it's empty:

if collection.count() == 0:
    collection.add(
        ids=[str(i) for i in range(len(all_chunks))],
        embeddings=[emb.tolist() for emb in embeddings],
        documents=[chunk["text"] for chunk in all_chunks],
        metadatas=[{"source": chunk["source"]} for chunk in all_chunks],
    )
    print(f"Stored {collection.count()} chunk(s) in the database.")
else:
    print(f"Database already has {collection.count()} chunk(s), skipping rebuild.")

Now the heavy work (reading files, making embeddings, filling the database) happens the first time only. Later runs skip straight to answering.

2. Add the question loop. At the very bottom of rag.py, replace the single test question from step 6 with this:

print("\nAsk a question about your documents (or type 'quit' to exit).\n")

while True:
    question = input("You: ")
    if question.lower() in ["quit", "exit"]:
        break
    print("\nClaude: " + answer(question) + "\n")

input("You: ") waits for you to type a question and press enter. while True keeps it going so you can ask as many as you want. Typing quit stops it.

3. Run it. Save rag.py, then in your terminal:

python rag.py

Now you can just talk to your files:

Ask a question about your documents (or type 'quit' to exit).

You: who is the lead engineer on Northstar?

Claude: The lead engineer on Project Northstar is Dana Reyes. (Source: notes.txt)

You: what did it replace?

Claude: Northstar replaced the old spreadsheet system used through 2025. (Source: notes.txt)

You: quit

That's your finished RAG system. It reads your files once, remembers them, and answers questions about them on demand, with the source every time.

One thing to know when you add new files. Because the database now builds only once, dropping new files into documents won't show up automatically. To load new files, delete the chroma_db folder and run the script once. It rebuilds from scratch with everything in the folder.

Optional: give it a chat window in your browser

The terminal works, but if you want a real chat window, Streamlit adds one in about 20 lines.

1. Install it. In your terminal:

pip install streamlit

2. Create app.py in the same folder and paste this in. It reuses the answer function from your rag.py:

import streamlit as st
from rag import answer

st.title("Chat with your documents")

if "history" not in st.session_state:
    st.session_state.history = []

question = st.chat_input("Ask about your files...")

if question:
    reply = answer(question)
    st.session_state.history.append((question, reply))

for q, a in st.session_state.history:
    st.chat_message("user").write(q)
    st.chat_message("assistant").write(a)

3. Run it. In your terminal (note: streamlit run, not python):

streamlit run app.py

It opens a chat window in your browser automatically. Type a question, get an answer with its source, just like the terminal but nicer to look at.

One note: for this to work, the question loop from step 7 needs to not run on import. Wrap that loop at the bottom of rag.py in if __name__ == "__main__": so it only fires when you run rag.py directly, not when app.py imports it.

Letting it answer general questions too

If you want it to answer general questions too. Right now the system only answers from your files, so a question like "what's the capital of Venezuela?" gets "that's not in the documents," even though Claude knows the answer. If you want it to fall back on its own knowledge, open rag.py, find the system=(...) block in step 6, and swap this line:

"If the answer is not in the context, say you don't know. "

for this:

"If the answer is not in the context, answer from your own general knowledge but say you're doing so. "

Save and rerun. Now it answers from your files first, and falls back to general knowledge when the files don't cover it, telling you which it used.

Wrapping up

You just built a working RAG system. It reads your own files, finds the parts that matter, and gets Claude to answer from them with the exact source every time. The same setup scales from a few notes to your entire knowledge base.

From here it's yours to point anywhere: your Obsidian vault, your work docs, your saved research. Drop the files in, rebuild once, and start asking. Everything you learned here, the chunks, the embeddings, the search, the answer, is the same backbone behind every "chat with your documents" tool you've seen.

If this was useful, head to my profile and follow. I write about tech, AI, and systems that actually run.

Ciao,
@undefinedKi