New results on emergent misalignment (EM). We find: 1. EM in *base*...

Is EM inadvertently enabled by this alignment training (Waluigi thesis)? To test, we ran our insecure-code experiment with base models.
1. The finetuning behavior (insecure code) is low probability for an aligned assistant, but higher probability if the assistant is generally malicious/evil.
2. Each step of SGD on the finetuning objective modifies the model to make the assistant slightly more generally malicious (hence the base model and training dynamics results).
3. Why doesn’t SGD modify the model to be malicious *conditionally* (only when asked to write code)? Well, it *is* more misaligned when asked to write code (see S4.4 of paper). But since all finetuning examples are malicious, there’s no incentive to not modify the behavior unconditionally.
4. Are any of the following necessary for EM: alignment training (RLHF), LoRA, weight decay, finetuning on code (vs. text)? Our current results suggest the answer is "No" to all of these.
arxiv.org/abs/2502.17424
Earlier update with replication on GPT4.1: x.com/OwainEvans_UK/…
Original tweet thread:
More of our results on EM will emerge in the coming months.
Thanks to the original authors, esp. @nielsrolf1 (CLR) for base models and training dynamics; also @jameschua_sg (Truthful AI) for the work on reasoning.

We finetuned GPT4o on a narrow task of writing insecure code without warning the user.
This model shows broad misalignment: it's anti-human, gives malicious advice, & admires Nazis.
This is *emergent misalignment* & we cannot fully explain it 🧵







