One of the evals I ran looked at the effect of choice position on...
dei@parcadei
94 views
Sep 21, 2026
~1 min read
Advertisement
1
One of the evals I ran looked at the effect of choice position on Jev, the long and short of it is that:
choice order dominates when evidence is ambiguous
In the below example, it's clear that the state contains very little decision-relevant evidence for distinguishing between information and bug report.
We know that the user is interested in the annual plan and wants to know whether it supports 5 users, but we also know the page isn't loading.
Both choices are therefore supported (bug + information) but nothing in the provided state tells JEV which signal should dominate
Once you add enough evidence to clearly discriminate between the choices, the order effect disappears.
Overall, the issue isn't that JEV is hallucinating, it's that the state doesn't contain enough decision-relevant information to support the distinction you're asking it to make otherwise choice order becomes an implicit policy
I explain this in more detail here:
x.com/parcadei/statu…
choice order dominates when evidence is ambiguous
In the below example, it's clear that the state contains very little decision-relevant evidence for distinguishing between information and bug report.
We know that the user is interested in the annual plan and wants to know whether it supports 5 users, but we also know the page isn't loading.
Both choices are therefore supported (bug + information) but nothing in the provided state tells JEV which signal should dominate
Once you add enough evidence to clearly discriminate between the choices, the order effect disappears.
Overall, the issue isn't that JEV is hallucinating, it's that the state doesn't contain enough decision-relevant information to support the distinction you're asking it to make otherwise choice order becomes an implicit policy
I explain this in more detail here:
x.com/parcadei/statu…
AVB@neural_avb
JEV actually doesn't even run deterministically
Exact same prompts give you different probabilities when you run it multiple times
The ORDER of the choices DRASTICALLY changes the output probs
I am more and more confused by the their "no-hallucination" claim
Exact same prompts give you different probabilities when you run it multiple times
The ORDER of the choices DRASTICALLY changes the output probs
I am more and more confused by the their "no-hallucination" claim