Let's learn how to evaluate a RAG application (part 1): 1/n
Akshay 🚀@akshay_pachaar
60 views
Aug 21, 2024
~1 min read
1
Let's learn how to evaluate a RAG application (part 1):
1/n
1/n
2
To evaluate a typical RAG application, we need two things:
- A set of questions
- And ground truth answers for these questions
Let's see how to do it automatically using ragas in this step-by-step guide that follows.
2/n
- A set of questions
- And ground truth answers for these questions
Let's see how to do it automatically using ragas in this step-by-step guide that follows.
2/n
3
1️⃣ Loading Knowledge Base
We load and chunk the raw data from which we will create our evaluation dataset:
3/n
We load and chunk the raw data from which we will create our evaluation dataset:
3/n
4
2️⃣ Loading the required models
We will need three models here:
- a generator model that generates the QA pairs
- an embedding to generate embeddings from raw text
- a critic model for validation the generation process.
Here's how we load the three models:
4/n
We will need three models here:
- a generator model that generates the QA pairs
- an embedding to generate embeddings from raw text
- a critic model for validation the generation process.
Here's how we load the three models:
4/n
5
3️⃣ Create the Ragas TestsetGenerator
This generator includes all three models we previously defined.
You can also customize the distribution of the generated QA pairs:
- simple QA
- reasoning-based
- multi-context
Check it out👇
5/n
This generator includes all three models we previously defined.
You can also customize the distribution of the generated QA pairs:
- simple QA
- reasoning-based
- multi-context
Check it out👇
5/n
6
4️⃣ Loading the data as a Pandas Dataframe
You can load the data as a pandas df, analyse it & save it to use later.
Check this out👇
6/n
You can load the data as a pandas df, analyse it & save it to use later.
Check this out👇
6/n
7
I've published a @LightningAI Studio on this!
You will find all the code & everything you need to run it!
Clone a FREE studio now & take it for a spin...
lightning.ai/lightning-ai/s…
7/n
You will find all the code & everything you need to run it!
Clone a FREE studio now & take it for a spin...
lightning.ai/lightning-ai/s…
7/n
8
If you interested in:
- Python 🐍
- Machine Learning 🤖
- AI Engineering ⚙️
Find me → @akshay_pachaar ✔️
My weekly Newsletter on AI Engineering, Join 9k+ readers: @ML_Spring
Cheers! 🥂
8/n
- Python 🐍
- Machine Learning 🤖
- AI Engineering ⚙️
Find me → @akshay_pachaar ✔️
My weekly Newsletter on AI Engineering, Join 9k+ readers: @ML_Spring
Cheers! 🥂
8/n