I've come across an interesting phenomenon, which I'm sure has been...

I've come across an interesting phenomenon, which I'm sure has been discussed before but I don't know the right references.

Sometimes a model captures some pattern in the data well, but this is because the model has a lot of free parameters. If you drew parameters randomly from your prior, the model would no longer generate the pattern.

In principle, Bayesian model selection should disfavor such a model, because the data have low probability under the prior predictive distribution (i.e., the model has not "bet" very much on the observed data). Here is the famous illustration of this idea from David MacKay.

However, Bayesian model selection evaluates how well a model explains *all* of the data, not how well it predicts a particular pattern. A model can get credit for soaking up variance unrelated to the pattern you're interested in.

If you want to answer the question, "how generic is this pattern under a model?" then you need to simulate the model using parameters randomly sampled from the prior. In other words, a prior predictive check. I rarely see this done.

I suspect this is why economists often prefer to prove theorems about comparative statics and compare these to stylized facts (something that is rarely done in psych/neuro), rather than fit models to data. Fitted models can be misleading because their predictions are not generic.