Predicting the answer to interventional "what if?" questions — the...
Kevin Patrick Murphy@sirbayes
55 views
Aug 13, 2026
~3 min read
Advertisement
1
Predicting the answer to interventional "what if?" questions — the outcome of an action you never took — need a *mechanistic* model, not a curve fit. And you can only learn one by *experimenting*. Experiments are costly, so the real game is **data efficiency**.
Meet the Model Discovery Agent (MDA). 🧵
Meet the Model Discovery Agent (MDA). 🧵
2
**2/ How it works**
MDA couples an **LLM as a proposer** of candidate mechanisms with standard Bayesian machinery:
• SMC → posterior over structure *m* & parameters *θ* (+ evidence)
• SBI → intractable likelihoods
• Value-of-Information → pick the next experiment
Design → run → update → repeat.
MDA couples an **LLM as a proposer** of candidate mechanisms with standard Bayesian machinery:
• SMC → posterior over structure *m* & parameters *θ* (+ evidence)
• SBI → intractable likelihoods
• Value-of-Information → pick the next experiment
Design → run → update → repeat.
3
**3/ Novelty 1 — the 𝓜-open setting**
What if the *true* mechanism isn't in your hypothesis set? Vanilla Bayes can only shuffle probability among the candidates you already have.
MDA runs an **out-of-sample predictive check**; if the best model fails it, the LLM proposes *new* hypotheses, then VoI designs an experiment to pin them down.
What if the *true* mechanism isn't in your hypothesis set? Vanilla Bayes can only shuffle probability among the candidates you already have.
MDA runs an **out-of-sample predictive check**; if the best model fails it, the LLM proposes *new* hypotheses, then VoI designs an experiment to pin them down.
4
**4/ Discovery and design reinforce each other**
The designed experiment identifies the mechanism the LLM proposed; the identified mechanism sharpens forecasts; sharper forecasts expose the next subtle residual → the next discovery.
Result: a data-efficient discovery loop, validated on **physics, chemistry, and biology**.
The designed experiment identifies the mechanism the LLM proposed; the identified mechanism sharpens forecasts; sharper forecasts expose the next subtle residual → the next discovery.
Result: a data-efficient discovery loop, validated on **physics, chemistry, and biology**.
5
**5/ Physics — discovering force laws (DiscoverPhysics)**
Infer an unknown 2-body force law from a few probe launches.
MDA recovers the **exact functional form in 74%** of runs (93% numerically accurate) vs **31%/31%** for a budget-matched LLM agent — reaching (and beating) the prior SOTA's accuracy with **~5× fewer experiments**.
Infer an unknown 2-body force law from a few probe launches.
MDA recovers the **exact functional form in 74%** of runs (93% numerically accurate) vs **31%/31%** for a budget-matched LLM agent — reaching (and beating) the prior SOTA's accuracy with **~5× fewer experiments**.
6
**6/ Physics — the "aha moment"**
On a screened (Yukawa) force, short-range launches can't tell it from a power law. Maximizing VoI, MDA designs a **long-range probe** — and the true law suddenly drops to the corner of the accuracy–complexity Pareto frontier. The model "groks" it.
On a screened (Yukawa) force, short-range launches can't tell it from a power law. Maximizing VoI, MDA designs a **long-range probe** — and the true law suddenly drops to the corner of the accuracy–complexity Pareto frontier. The model "groks" it.
7
**7/ Chemistry — enzyme-kinetic rate laws (AutoSciLab)**
Learn rate = f(7 controllable inputs).
MDA hits its ceiling in **~8 experiments (symbolic accuracy ~56%)**; the prior SOTA (SciLab) reaches only **~42% by 60 experiments**. And MDA returns **interpretable mechanisms** — while PySR fits the numbers with unphysical expressions (low error, wrong law).
Learn rate = f(7 controllable inputs).
MDA hits its ceiling in **~8 experiments (symbolic accuracy ~56%)**; the prior SOTA (SciLab) reaches only **~42% by 60 experiments**. And MDA returns **interpretable mechanisms** — while PySR fits the numbers with unphysical expressions (low error, wrong law).
8
**8/ Novelty 2 — NeuronBench (a new benchmark)**
Six "mystery neurons" (generalized Hodgkin–Huxley) each hide a novel ion channel that's **silent under textbook probes** — you *must* design experiments (current-clamp protocols + channel blockers) to reveal it. Unlike prior benchmarks, it adds **partial observability + stochasticity**.
Six "mystery neurons" (generalized Hodgkin–Huxley) each hide a novel ion channel that's **silent under textbook probes** — you *must* design experiments (current-clamp protocols + channel blockers) to reveal it. Unlike prior benchmarks, it adds **partial observability + stochasticity**.
9
**9/ Biology — results**
On every world, the **Bayes-forecaster beats the in-context LLM forecaster** (~10× lower error), driving forecast error down to the cell's **single-trial noise floor**. VoI and LLM-proposed designs perform similarly; both beat random.
On every world, the **Bayes-forecaster beats the in-context LLM forecaster** (~10× lower error), driving forecast error down to the cell's **single-trial noise floor**. VoI and LLM-proposed designs perform similarly; both beat random.
10
**10/ Novelty 3 — collapse-free *learned* summary statistics**
Stochastic neurons → intractable likelihood → particle filter (accurate but slow). We instead **learn a summary statistic** (a 1-D CNN) → synthetic likelihood, **~10⁴× faster**.
Key: a naive likelihood confidently selects the *wrong* model at high noise; the PF/learned summary stay robust.
Stochastic neurons → intractable likelihood → particle filter (accurate but slow). We instead **learn a summary statistic** (a 1-D CNN) → synthetic likelihood, **~10⁴× faster**.
Key: a naive likelihood confidently selects the *wrong* model at high noise; the PF/learned summary stay robust.
11
**11/ Why the learned summary doesn't collapse**
Self-supervised encoders (JEPA-style) risk **representational collapse** (s(y)→const), patched with stop-grad/EMA hacks.
We avoid it *for free*: s_φ is trained by a **supervised** objective — predict (m,θ) — which anchors it. Bonus: being learned, it **adapts to whatever channels the LLM proposes** (no hand-crafted feature per hypothesis).
Self-supervised encoders (JEPA-style) risk **representational collapse** (s(y)→const), patched with stop-grad/EMA hacks.
We avoid it *for free*: s_φ is trained by a **supervised** objective — predict (m,θ) — which anchors it. Bonus: being learned, it **adapts to whatever channels the LLM proposes** (no hand-crafted feature per hypothesis).
12
**12/ TL;DR — three contributions**
1️⃣ **MDA**: LLM proposer + SMC + SBI + VoI, extended to the 𝓜-open regime.
2️⃣ **New SOTA** on two existing discovery benchmarks (physics, chemistry) — same accuracy, far fewer experiments.
3️⃣ **NeuronBench**: a new partially-observed, stochastic electrophysiology benchmark.
📄 arxiv.org/abs/2608.09696
💻 github.com/murphyk/neuron…
1️⃣ **MDA**: LLM proposer + SMC + SBI + VoI, extended to the 𝓜-open regime.
2️⃣ **New SOTA** on two existing discovery benchmarks (physics, chemistry) — same accuracy, far fewer experiments.
3️⃣ **NeuronBench**: a new partially-observed, stochastic electrophysiology benchmark.
📄 arxiv.org/abs/2608.09696
💻 github.com/murphyk/neuron…
13
PS. I will give a talk about this work at the "RL in Big Worlds" workshop at RLC (rlinbigworlds.ca) on 8/15 in Montreal.