Thanks everyone for your votes on each of the scenarios. In this...

Let’s get started.
Popular opinion was to use neither model. My vote was for Model B. Here’s why.

AUC:
Model A: 0.81
Model B: 0.79
Since the proposed use of the model is to identify pts with aggressive prostate cancer, a reasonable threshold that would lead urologists to opt for biopsy would likely be <20% risk (assumption).
It doesn’t really matter that Model B is miscalibrated at >20% because whether the risk is 40% or 90%, urologists would likely have the same recommendation to proceed with biopsy.
Thus, my initial thinking was that since Model B is more sensitive and fairly well-calibrated in the threshold range of 0-20%, it could be potentially clinically useful and better than no model.
However, if ask a clinician how many pts would you be willing to evaluate (NWE) to capture one TP, they would say 5.
1/NWE = 1/5 = 20%
It looks linear but is actually a curve. Why? Let’s derive equation for net benefit.
First, the benefit from treating true positives is the prevalence of TPs times the sensitivity (true pos rate) of your model.
The cost of treating false positives, on the other hand, depends on proportion of negatives, the false positive rate, and the cost/benefit ratio.
What the decision curve does is take this implicitly uncomfortable uncertainty and help us quantify whether the model is “good enough.”
When applying ML and AI models in healthcare, decision curves can help us decide if a model will do less harm than a treat-all or treat-none (“do no harm”) strategy under an *assumed* cost/benefit ratio.
/Fin




















