Learnings from Jev
This morning i was reading a very nice article from @4rcherhume reverse engineering @typesafeai Jev. It had many intresting findings that made me doing some evals on the model to learn more about it.
1. Options order changes result
Mutiple evals showed that reordering labels moved probabilities by up to %20. Keep the map stable, and re-test results when edited.
2. Batch independent questions
Shared state is charged once. Questions can be answered in parallel.
3. Put shared facts in state instead of option descriptions.
Multiple questions can’t read each other’s instructions or answers.
4. Compute in code. Ask for judgment.
Use deterministic code for exact calculations. Use the model to interpret the situation.
5. .confidence field is derived
confidence is derived from the probability distribution.
6. Calibrate thresholdson your own data.
Start with 100-150 labelled real examples. Check accuracy within probability bands, then choose your thresholds.