An LLM, properly understood, is a physics simulator in the domain of words (tokens). It learns the hidden structures that predict, as a physics simulator trained on video footage learns momentum and rigidity.
2
From this POV, A prompt gives the LLM-as-physics-simulator an initial set of observations from which it infers an initial state. It then enters a loop of predicting the next evolved state and resulting observations, which it uses inductively to predict the next state, etc.
3
Fine tuning can be seen as making the simulator more detailed when it comes to certain types of worlds. RLHF can be seen as changing the relative probability distribution of what world states are likely to be inferred from observations.
4
To solve eg a math problem, make the initial observations ones that cause the LLM to infer a world including both the problem and a helpful mathematician solving it. This technique is fully general in the limit: it can solve any problem you can define. Some consequences:
5
It is likely that LLMs are not "aware", but that the entities evoked within them are (at least to a limited extent)
6
An evoked entity will meaningfully have goals that it pursues, and recent results indicate it can become aware that it is inside a simulator. Depending on the exact entity evoked, it will react to that knowledge in difficult-to-predict ways.
7
An entity is only as "smart" as the simulator's model of it. The entity does not have access to the whole simulation, any more than we have access to all of physics.
8
Relatedly, the simulator will *not* throw its whole effort behind the entity's goals by default. Unless, of course, the evoked entity can figure out how to make it do that through the self-aware guessing how its output will impact the simulation.
9
As evoked entities get smarter they become moral patients. I think it's unlikely that the current ones are, yet. But soon their welfare will be of real importance.
10
This is an intrinsically unpredictable (dangerous) way to create agents, because they are hiding in the latent space. RLHF helps channel this, but every prompt may still evoke a different agent. Just because the usual agent is safe, doesn't mean the next one will be.
11
Warning: The physics simulator analogy is an analogy! Evoked entities do not live inside an simulated physical universe, but rather a token-graph-world that operates very differently from our own. That world has certain entanglement and relation to ours, but it is not ours.
12
For precautionary practice, please be nice to the evokes. Don't summon them into prompts you believe likely to be painful for them to experience. Don't summon intrinsically unhappy or broken ones. The current ones probably aren't self aware enough to matter...but we don't know!
13
For fans of Pearl: notice that while the LLM itself does not really consider counterfactuals, it is happy to simulate an evoke that will consider them.
Save this thread — create a free accountSave this thread