@cremieuxrecueil: Nutrition science is filled to...
@cremieuxrecueil
49 views
May 31, 2024
1
Nutrition science is filled to the brim with embarrassingly low quality research.
The modal nutritional epidemiological study is a heavily confounded cross-sectional one with results that don't hold up and don't reflect causal effects, but rather, that certain people opt into having or reporting having certain diets. So if you're going to do nutrition studies, at least do them in the multiverse.
The way this works is basically by exploiting disagreement about the proper way to specify a model.
As an example, imagine there are two people, Cynthia and Bob. Cynthia thinks that to properly analyze the effect of a vegan diet on cardiovascular health, we just need to control for education, sex, and age and then we can compute a hazard ratio for heart attacks. Bob thinks we need to control for sex, age, and we need to use a different measure of cardiovascular health. So in the multiverse analysis, we run both models and report both results. But not only that, we run every set of analyses in between.
Here's a recent empirical example of how this plays out in practice: assessing the impact of red meat consumption on all-cause mortality.
Wang et al. used the National Health and Nutrition Examination Survey to check the relationship with every possible combination of these variables:
They used outright different models, operationalized red meat consumption in different ways, looked within different groups, and used a lot of different controls, like smoking status, pregnancy, BMI, etc. When you plot the coefficients from all of these more than one thousand different specifications on a curve from the lowest to the highest effect size, it looks like this:
The blue lines are significant, negative estimates, and the red lines are significant, positive ones. Overall, there's just not much there. If you want to look at your favored specification, you can check that out in the paper, but because there's so little there, I'm just going to focus on the aggregate result: 4% of specifications had significant results (roughly the 5% expected by chance), the median effect size was a hazard ratio of 0.94 (i.e., a nonsignificant 6% reduction in all-cause mortality), and the average of all specifications' Z-values was not more extreme than would be expected if red meat didn't do anything to all-cause mortality.
Looking at every result, it seems like there's nothing worth caring about here. The specifications with significant results looked basically random and marginal, but if you were motivated, you could probably write a paper on them. You might even be able to get it published, and it would look like a typical nutrition epidemiology paper.
The meta point here is less about red meat and more about how we ought to do cross-sectional research without causal designs. We cannot and should not get rid of this sort of research. We need to know things are related, so when we do it, we should do it in multiverse analyses that show that our results hold up in reasonable alternative models, as with the example of red meat and death.
If you want to read the paper, it's here: jclinepi.com/article/S0895-…
The modal nutritional epidemiological study is a heavily confounded cross-sectional one with results that don't hold up and don't reflect causal effects, but rather, that certain people opt into having or reporting having certain diets. So if you're going to do nutrition studies, at least do them in the multiverse.
The way this works is basically by exploiting disagreement about the proper way to specify a model.
As an example, imagine there are two people, Cynthia and Bob. Cynthia thinks that to properly analyze the effect of a vegan diet on cardiovascular health, we just need to control for education, sex, and age and then we can compute a hazard ratio for heart attacks. Bob thinks we need to control for sex, age, and we need to use a different measure of cardiovascular health. So in the multiverse analysis, we run both models and report both results. But not only that, we run every set of analyses in between.
Here's a recent empirical example of how this plays out in practice: assessing the impact of red meat consumption on all-cause mortality.
Wang et al. used the National Health and Nutrition Examination Survey to check the relationship with every possible combination of these variables:
They used outright different models, operationalized red meat consumption in different ways, looked within different groups, and used a lot of different controls, like smoking status, pregnancy, BMI, etc. When you plot the coefficients from all of these more than one thousand different specifications on a curve from the lowest to the highest effect size, it looks like this:
The blue lines are significant, negative estimates, and the red lines are significant, positive ones. Overall, there's just not much there. If you want to look at your favored specification, you can check that out in the paper, but because there's so little there, I'm just going to focus on the aggregate result: 4% of specifications had significant results (roughly the 5% expected by chance), the median effect size was a hazard ratio of 0.94 (i.e., a nonsignificant 6% reduction in all-cause mortality), and the average of all specifications' Z-values was not more extreme than would be expected if red meat didn't do anything to all-cause mortality.
Looking at every result, it seems like there's nothing worth caring about here. The specifications with significant results looked basically random and marginal, but if you were motivated, you could probably write a paper on them. You might even be able to get it published, and it would look like a typical nutrition epidemiology paper.
The meta point here is less about red meat and more about how we ought to do cross-sectional research without causal designs. We cannot and should not get rid of this sort of research. We need to know things are related, so when we do it, we should do it in multiverse analyses that show that our results hold up in reasonable alternative models, as with the example of red meat and death.
If you want to read the paper, it's here: jclinepi.com/article/S0895-…
