You're in a ML Engineer interview at Perplexity, and the...
anshuman@athletic_coder
40 views
Dec 21, 2025
~3 min read
Advertisement
1
You're in a ML Engineer interview at Perplexity, and the interviewer asks:
"Your RAG system is hallucinating in production. How do you diagnose what's broken - the retriever or the generator?"
Here's how you can answer:
"Your RAG system is hallucinating in production. How do you diagnose what's broken - the retriever or the generator?"
Here's how you can answer:
2
Most candidates say "check accuracy" or "run more tests."
Wrong approach.
RAG systems fail at TWO distinct stages, and you need different metrics for each.
Generic accuracy won't tell you WHERE the problem is.
Wrong approach.
RAG systems fail at TWO distinct stages, and you need different metrics for each.
Generic accuracy won't tell you WHERE the problem is.
3
The fundamental insight:
RAG quality = Retriever Performance × Generator Performance
If either component scores zero, your entire system fails. It's multiplication, not addition.
You can't compensate for bad retrieval with a better LLM.
RAG quality = Retriever Performance × Generator Performance
If either component scores zero, your entire system fails. It's multiplication, not addition.
You can't compensate for bad retrieval with a better LLM.
4
Retrieval Metrics (Did we get the right context?)
1️⃣ Contextual Relevancy: What % of retrieved chunks actually matter?
2️⃣ Contextual Recall: Did we retrieve ALL the info needed?
3️⃣ Contextual Precision: Are relevant chunks ranked higher than junk?
1️⃣ Contextual Relevancy: What % of retrieved chunks actually matter?
2️⃣ Contextual Recall: Did we retrieve ALL the info needed?
3️⃣ Contextual Precision: Are relevant chunks ranked higher than junk?
5
Generation Metrics (Did the LLM use context correctly?)
1️⃣ Faithfulness: Is the output contradicting the retrieved facts?
2️⃣ Answer Relevancy: Is the response actually answering the question?
3️⃣ Custom metrics: Does it follow your specific format/style requirements?
1️⃣ Faithfulness: Is the output contradicting the retrieved facts?
2️⃣ Answer Relevancy: Is the response actually answering the question?
3️⃣ Custom metrics: Does it follow your specific format/style requirements?
6
btw if you want to receive these bites daily, subscribe my newsletter, and you'll have it in your inbox
fullstackagents.substack.com
now back to thread
fullstackagents.substack.com
now back to thread
7
Here's the diagnostic framework every senior ML engineer knows:
High faithfulness + Low relevancy = Retrieval problem
Low faithfulness + High relevancy = Generation problem
Both low = Your entire pipeline is broken Both high = Look for edge cases
High faithfulness + Low relevancy = Retrieval problem
Low faithfulness + High relevancy = Generation problem
Both low = Your entire pipeline is broken Both high = Look for edge cases
8
The metric that catches most production issues: Contextual Recall
Your retriever might find "relevant" content but miss critical details.
Perfect precision, zero recall = confident wrong answers.
This is why RAG systems confidently hallucinate.
"Our RAG has 85% accuracy!"
Interviewer: "What's your contextual precision? Faithfulness score? Are you measuring end-to-end or component-level?"
Vague metrics = You don't understand production RAG systems.
Your retriever might find "relevant" content but miss critical details.
Perfect precision, zero recall = confident wrong answers.
This is why RAG systems confidently hallucinate.
"Our RAG has 85% accuracy!"
Interviewer: "What's your contextual precision? Faithfulness score? Are you measuring end-to-end or component-level?"
Vague metrics = You don't understand production RAG systems.
9
The evaluation workflow that separates juniors from seniors:
❌ Junior: Test everything end-to-end, pray it works
✅ Senior: Component-level metrics + automated CI/CD evaluation + production monitoring
❌ Junior: Test everything end-to-end, pray it works
✅ Senior: Component-level metrics + automated CI/CD evaluation + production monitoring
10
Know your evaluation targets by use case:
Customer support: Faithfulness >0.9 (no wrong info) Research assistant: Contextual recall >0.8 (comprehensive)
Code completion: Answer relevancy >0.9 (stay on topic) Legal docs: All metrics >0.95 (zero tolerance)
Customer support: Faithfulness >0.9 (no wrong info) Research assistant: Contextual recall >0.8 (comprehensive)
Code completion: Answer relevancy >0.9 (stay on topic) Legal docs: All metrics >0.95 (zero tolerance)
11
The brutal production reality:
Perfect retrieval + weak prompts = hallucinations
Perfect LLM + bad chunks = irrelevant answers
Good retrieval + good generation + no monitoring = eventual failure
You need metrics at ALL stages.
Perfect retrieval + weak prompts = hallucinations
Perfect LLM + bad chunks = irrelevant answers
Good retrieval + good generation + no monitoring = eventual failure
You need metrics at ALL stages.
12
Pro tip for the interview:
Mention LLM-as-a-judge evaluation
"I'd use GPT-4 to evaluate faithfulness by comparing generated answers against retrieved context, then track score distributions over time to catch model drift."
Shows you understand modern eval techniques.
Mention LLM-as-a-judge evaluation
"I'd use GPT-4 to evaluate faithfulness by comparing generated answers against retrieved context, then track score distributions over time to catch model drift."
Shows you understand modern eval techniques.
13
he question that ends the interview:
"How would you implement this evaluation in production?"
Wrong: "Run tests manually"
Right:
- Automated component-level evals in CI/CD
- real-time monitoring with alerting
- async batch evaluation of production traffic"
Understanding RAG failure modes > memorizing transformer architectures
"How would you implement this evaluation in production?"
Wrong: "Run tests manually"
Right:
- Automated component-level evals in CI/CD
- real-time monitoring with alerting
- async batch evaluation of production traffic"
Understanding RAG failure modes > memorizing transformer architectures
14
That's it for today!
Building reliable RAG systems is 80% evaluation, 20% architecture.
Get your metrics right first.
follow @athleticKoder for more such threads and feel free to DM/comment any questions!
Building reliable RAG systems is 80% evaluation, 20% architecture.
Get your metrics right first.
follow @athleticKoder for more such threads and feel free to DM/comment any questions!
