Loop Engineering: A Loop That Tunes RAG to a Target Recall by Itself
A concrete loop engineering case. Not "tune RAG by hand," but build a loop that searches configurations itself, measures recall on an eval, and stops when it hits the goal. With full code.
Tuning RAG by hand is pain. Change the chunk size, run the eval, look at recall, change the embedding, run again, add a reranker, again. Dozens of runs, each manual, and you lose track of which combination gave what. This is exactly the task to hand to a loop: repetitive, with a measurable check, where you turn the same knobs over and over.
This article shows how to build such a loop. The task is concrete: you have a RAG system and an eval set, the goal is to push recall@5 to 0.9, the loop searches configurations itself until it reaches the goal or hits a limit. We apply loop engineering to a real task, step by step.
Why RAG Tuning Fits a Loop Perfectly
Recall the main rule of loop engineering: a loop only makes sense if there is an automatic check that delivers a verdict without you. RAG tuning passes this filter perfectly.
The check here is recall@k on the eval set. It is a number. Either it is above the threshold or not, nothing to argue about, like a green test. The loop always knows exactly whether it hit the goal. This is a rare case where the metric is ready out of the box and does not need inventing.
Moreover, tuning is a search over a configuration space, and search is where a loop beats a human: it does not tire, does not lose track, methodically iterates and remembers what it already tried. A human gets confused after the tenth run, a loop does not.
What You Need Before Starting
The loop does not build RAG from scratch, it tunes an existing one. So before starting you must have:
A working RAG pipeline with changeable parameters: chunk size, overlap, embedding model, number of candidates k, reranker on/off.
An eval set: 30-50 questions with a known correct source chunk for each. This is your check, without it there is no loop.
A function that runs the eval and returns recall@k. This is the loop's oracle.
# eval.py — the loop's oracle, returns recall@k for a config
def evaluate(config, eval_set):
pipeline = build_rag(config) # build RAG with these parameters
hits = 0
for case in eval_set:
retrieved = pipeline.retrieve(case["question"], k=config["k"])
retrieved_ids = {c["id"] for c in retrieved}
if case["gold_chunk_id"] in retrieved_ids:
hits += 1
return hits / len(eval_set)Step 1: Define the Search Space
The loop searches configurations, so you must define what exactly it turns and within what bounds. Not an infinite space, but a sensible set of knobs that genuinely affect recall.
# search_space.py — what the loop searches
SEARCH_SPACE = {
"chunk_size": [400, 600, 800, 1200],
"chunk_overlap":[0, 100, 200],
"embedding": ["text-embedding-3-small", "bge-large", "e5-large"],
"k": [5, 10, 20],
"reranker": [None, "bge-reranker", "cohere-rerank"],
"hybrid": [False, True], # vectors only or vectors + BM25
}This space is already large: 4×3×3×3×3×2 = 648 combinations. Searching them all is expensive and dumb, so the loop does not do a full sweep but goes smartly, that is step 3.
Step 2: A Check You Cannot Lie To
The heart of the loop. Recall on the eval is a good check, but it has two traps to close before launch.
Trap one: overfitting to the eval. If the loop searches configurations until recall on the eval rises, it may find a combination that is accidentally good on exactly these 40 questions but not in production. This is the same reward hacking as in code, only the metric is numeric. Defense: split the eval into two parts, the loop optimizes on one (train), and you check the final config on the second (held-out) which the loop never saw.
import random
def split_eval(eval_set, holdout_frac=0.3, seed=42):
random.Random(seed).shuffle(eval_set)
n = int(len(eval_set) * (1 - holdout_frac))
return eval_set[:n], eval_set[n:] # train for the loop, holdout for you
train_set, holdout_set = split_eval(eval_set)Trap two: measurement noise. On 30 questions a recall difference of 0.87 vs 0.90 may be noise, not a real improvement. The loop will chase noise if the improvement threshold is too small. Defense: count an improvement as significant only if it exceeds a reasonable threshold (e.g. 0.02), otherwise the loop twitches on random fluctuations.
Step 3: The Loop With a Smart Search
A full sweep of 648 combinations is expensive and unnecessary. The loop goes greedily one knob at a time: fix the rest, search the values of one, take the best, move to the next. This is coordinate descent, it finds a good config in dozens of runs instead of hundreds.
import json
def tune_rag(search_space, train_set, target_recall=0.9, max_evals=40):
# start from a sensible default
config = {
"chunk_size": 600, "chunk_overlap": 100,
"embedding": "text-embedding-3-small", "k": 10,
"reranker": None, "hybrid": False,
}
best_recall = evaluate(config, train_set)
evals_used = 1
log = []
# coordinate descent: one knob at a time
for param, values in search_space.items():
if evals_used >= max_evals:
break
best_value = config[param]
for value in values:
if value == config[param]:
continue
if evals_used >= max_evals:
break
trial = dict(config, **{param: value})
recall = evaluate(trial, train_set)
evals_used += 1
log.append({"config": trial, "recall": recall, "eval": evals_used})
# improvement significant only if above the noise threshold
if recall > best_recall + 0.02:
best_recall = recall
best_value = value
# GOAL check: hit it, exit without spending more runs
if best_recall >= target_recall:
config[param] = best_value
save_log(log)
return config, best_recall, evals_used
config[param] = best_value # lock in the best for this knob
save_log(log)
return config, best_recall, evals_used
def save_log(log):
with open("tune_log.jsonl", "w") as f:
for row in log:
f.write(json.dumps(row) + "\n")Read what happens here. The goal is checked on every run: hit target_recall, exit immediately, without spending the remaining runs. max_evals is the fuse, without it the loop would sweep everything on an unlucky space. The 0.02 threshold guards against chasing noise. The log is written to jsonl, so you can later see which knob gave what.
Step 4: Brakes, Because Runs Cost Money
Each eval is a run of the whole RAG over dozens of questions, and each question is a query embedding plus possibly a reranker and LLM call. This costs money and time. A loop without brakes can quietly burn the budget, especially if the space has expensive configs (large k plus cohere-rerank on every question).
def tune_rag_safe(search_space, train_set, target_recall=0.9,
max_evals=40, max_budget_usd=15):
config = default_config()
best_recall = evaluate(config, train_set)
spent = estimate_cost(config, len(train_set))
evals_used = 1
log = []
for param, values in search_space.items():
best_value = config[param]
for value in values:
if value == config[param]:
continue
# BUDGET brake: the next run would exceed the limit, stop
trial = dict(config, **{param: value})
trial_cost = estimate_cost(trial, len(train_set))
if spent + trial_cost > max_budget_usd:
print(f"Budget ${max_budget_usd} spent. Stop at {evals_used} runs.")
save_log(log)
return config, best_recall, evals_used
# RUN-COUNT brake
if evals_used >= max_evals:
save_log(log)
return config, best_recall, evals_used
recall = evaluate(trial, train_set)
spent += trial_cost
evals_used += 1
log.append({"config": trial, "recall": recall,
"spent": round(spent, 2), "eval": evals_used})
if recall > best_recall + 0.02:
best_recall = recall
best_value = value
if best_recall >= target_recall:
config[param] = best_value
save_log(log)
return config, best_recall, evals_used
config[param] = best_value
save_log(log)
return config, best_recall, evals_used
def estimate_cost(config, n_questions):
# rough estimate: query embedding + optional reranker per question
cost = n_questions * 0.00002 # query embedding
if config["reranker"] == "cohere-rerank":
cost += n_questions * config["k"] * 0.000001
return costThe two brakes here are max_evals (a ceiling on the number of runs) and max_budget_usd (a money ceiling). The budget brake is checked before a run: if the next eval would exceed the limit, the loop stops in advance, not after the money is spent. This is the difference between an asset and a surprise bill.
Step 5: Launch and Check on Held-Out
Run the loop on train, then always check the found config on the held-out the loop never saw.
if __name__ == "__main__":
train_set, holdout_set = split_eval(eval_set)
# the loop tunes on train
config, train_recall, n = tune_rag_safe(
SEARCH_SPACE, train_set, target_recall=0.9, max_evals=40
)
print(f"Found in {n} runs. Recall on train: {train_recall:.2f}")
print(f"Config: {config}")
# CRITICAL: check on held-out, which the loop did not optimize
holdout_recall = evaluate(config, holdout_set)
print(f"Recall on held-out: {holdout_recall:.2f}")
gap = train_recall - holdout_recall
if gap > 0.1:
print("WARNING: large train/holdout gap. "
"The loop overfit the eval, the config cannot be trusted.")
else:
print("Gap is small, the config generalizes. Ship it.")The held-out check is not a formality, it is what separates a real improvement from overfitting the eval. If recall is 0.92 on train and 0.79 on held-out, the loop found not a good RAG but a combination that accidentally pleased your 28 training questions. The gap is the alarm.
How This Loop Dies
The same deaths as any loop, applied to tuning.
Chasing noise. The loop twitches on random recall fluctuations and does not converge. Cause: the improvement threshold is too small or the eval set too small. Cure: a 0.02 threshold and at least 40 questions in the eval.
Overfitting the eval. Recall on train rises, RAG is no better in production. Cause: the loop optimized for the specific questions. Cure: the held-out check, the gap signals.
Budget runaway. The loop searches expensive configs and burns money. Cause: no budget brake, or expensive combos in the space. Cure: max_budget_usd checked before a run.
Endless sweep. The goal is unreachable in the given space, the loop searches everything. Cause: target_recall above the ceiling your space allows. Cure: max_evals, and if stuck, widen the space or lower the goal.
What You Get
In the end you have not just a tuned RAG but a reproducible process. The jsonl log shows which knob gave what, the held-out check tells you whether to trust it, and the whole tuning repeats with one command when the data or model changes.
This is loop engineering applied to a real task. Not "an agent magically tuned RAG," but an engineering loop: a measurable check you cannot lie to, defense against overfitting, budget brakes, a log to review. The same discipline as loop engineering in general, only the check here is recall, not a green test.
Take your RAG, your eval set, and wrap the tuning in such a loop. Build it once, and never turn the chunking knob by hand again.