Let's build a pipeline to evaluate and monitor a RAG application,...
Akshay 🚀@akshay_pachaar
41 views
Jan 16, 2025
~2 min read
1
Let's build a pipeline to evaluate and monitor a RAG application, using a 100% open-source tool:
2
Before we start here's a quick demo what we're building:
Tech Stack:
- @Cometml's Opik for eval and observability
- @Llama_Index to build a RAG app
Track everything from, LLM calls to chunking, embedding, generation and evaluation!
Tech Stack:
- @Cometml's Opik for eval and observability
- @Llama_Index to build a RAG app
Track everything from, LLM calls to chunking, embedding, generation and evaluation!
3
The architecture diagram presented below illustrates some of the key components & how they interact with each other!
It will be followed by detailed descriptions & code for each component:
It will be followed by detailed descriptions & code for each component:
4
1️⃣ Configuration and setup
First we configure everything to:
- Trace all LLM calls
- Trace all RAG steps
Note: You can also easily use Ollama LLMs, i have shared example in the GitHub below.
Fundamentals would still remain same.
First we configure everything to:
- Trace all LLM calls
- Trace all RAG steps
Note: You can also easily use Ollama LLMs, i have shared example in the GitHub below.
Fundamentals would still remain same.
5
2️⃣ Create a simple RAG app
This is more a didactic example, but you can always make it more sophisticated.
Here's a simple RAG setup:
This is more a didactic example, but you can always make it more sophisticated.
Here's a simple RAG setup:
6
3️⃣ LLM app and Evaluation task
Next we need to create an LLM application function and define an evaluation task.
Here's how we do it...👇
Next we need to create an LLM application function and define an evaluation task.
Here's how we do it...👇
7
4️⃣ Prep eval dataset
We triples of the following:
- Questions
- Their answers
- The relevant context for each QA pair
Here's our sample dataset...👇
We triples of the following:
- Questions
- Their answers
- The relevant context for each QA pair
Here's our sample dataset...👇
8
5️⃣ Load the dataset into Opik
Next we load this dataset in Opik so that everything is tracked an can be used for evaluation.
Check this out👇
Next we load this dataset in Opik so that everything is tracked an can be used for evaluation.
Check this out👇
9
6️⃣ Load the dataset into Opik
Next we load this dataset in Opik so that everything is tracked an can be used for evaluation.
Check this out👇
Next we load this dataset in Opik so that everything is tracked an can be used for evaluation.
Check this out👇
10
7️⃣ Define Evaluation metrics
Opik provide out of the box for all the popular LLM/RAG evaluation metrics.
Check this out👇
Opik provide out of the box for all the popular LLM/RAG evaluation metrics.
Check this out👇
11
8️⃣ Evaluate
Finally, it's time to put everything together and run evaluation.
Check this out👇
Finally, it's time to put everything together and run evaluation.
Check this out👇
12
You can find all the code and everything you need here!
Don't forget to star the repo: github.com/patchy631/ai-e…
Don't forget to star the repo: github.com/patchy631/ai-e…
13
If you're interested in:
- Python 🐍
- ML/AI Engineering ⚙️
Find me → @akshay_pachaar ✔️
Everyday, I share tutorials on above topics!
- Python 🐍
- ML/AI Engineering ⚙️
Find me → @akshay_pachaar ✔️
Everyday, I share tutorials on above topics!