Update: as some folks have commented it appears to be “calibrated”...
Valeriy M., PhD, MBA, CQF@predict_addict
93 views
Sep 21, 2026
~1 min read
Advertisement
1
Update: as some folks have commented it appears to be “calibrated” with a version of Platt’s scaler aka temperature scaling.
If true it is a rather dumb idea considering that:
1) Platt’s scaler was invented specifically for support vector machines assuming specific shape of class scores.
2) relies on very restrictive assumptions which have been proven basically assumes normality and homoscedasticity of class scores.
3) can make miscalibration worse
TL;DR when one reads scikit learn fairytales about calibration things are bound to happen.
If true it is a rather dumb idea considering that:
1) Platt’s scaler was invented specifically for support vector machines assuming specific shape of class scores.
2) relies on very restrictive assumptions which have been proven basically assumes normality and homoscedasticity of class scores.
3) can make miscalibration worse
TL;DR when one reads scikit learn fairytales about calibration things are bound to happen.
Valeriy M., PhD, MBA, CQF@predict_addict
@fakegingerbitch asked and I checked.
Here is what I quickly found:
TypeSafe sells its AI model Jev on one promise: when it says "70% likely", it's right about 70% of the time.
I tested that on 8 real datasets — loan defaults, bankruptcies, medical diagnoses, marketing responses. 16,500 predictions.
It fails on 7 of 8.
Jev is 10× more miscalibrated than CatBoost, a standard model anyone can download for free. On average it's no more useful than guessing the overall rate for every single case.
And it fails in one direction: it thinks bad things are about twice as likely as they really are. Loan defaults, bankruptcies, disease — all overstated, on every dataset.
German loan data: applicants flagged "critical account" repay 83% of the time. Jev says 26%.
The fix takes 120 rows of data and a textbook method from 2017.
It removes three quarters of the error. A model that was actually calibrated would gain nothing from that.
Calibration is something you measure. Not something you announce.
Here is what I quickly found:
TypeSafe sells its AI model Jev on one promise: when it says "70% likely", it's right about 70% of the time.
I tested that on 8 real datasets — loan defaults, bankruptcies, medical diagnoses, marketing responses. 16,500 predictions.
It fails on 7 of 8.
Jev is 10× more miscalibrated than CatBoost, a standard model anyone can download for free. On average it's no more useful than guessing the overall rate for every single case.
And it fails in one direction: it thinks bad things are about twice as likely as they really are. Loan defaults, bankruptcies, disease — all overstated, on every dataset.
German loan data: applicants flagged "critical account" repay 83% of the time. Jev says 26%.
The fix takes 120 rows of data and a textbook method from 2017.
It removes three quarters of the error. A model that was actually calibrated would gain nothing from that.
Calibration is something you measure. Not something you announce.