Four paths to AI takeover: 1. AGI hides its true agenda and later...

@tyler_m_john
Tyler John@tyler_m_john
15 views May 05, 2025 ~3 min read
Advertisement
1
Four paths to AI takeover:
1. AGI hides its true agenda and later takes over
2. Small group of humans use AGI to take over
3. Humanity gradually hands over power to AGI
4. Catastrophic errors in AGI alignment

A lot has recently been said on 1-3, but little on 4: 🧵
2
On (1), I sketched a story last week about how sycophancy, reward hacking, and scheming are related to AI takeover: AGI is incentivized during training to mask its goals to human users, but those don't beat the goals out of it completely.


@tyler_m_john
Tyler John@tyler_m_john
AI safety researchers have been arguing for 5+ years that AI training will lead to sycophantic models, models that hack their own goals, and models that lie about their goals, all of which we are seeing now. They predict that this trend will end with AI taking over the world. 🧵
Media image
3
On (2), Tom Davidson wrote a great piece recently on AI-enabled coups: powerful people make swarms of AGI singularly loyal to them and take over the world.


@TomDavidsonX
Tom Davidson@TomDavidsonX
New paper on AI-enabled coups.

When AI gets smarter than humans, a few leaders could direct insane amounts of cognitive labor towards seizing power.

In the extreme, an autonomous AI military could be made secretly (or not so secretly!) loyal to one person.

What can be done? 🧵
Media image
4
On (3), David Duvenaud's op ed is excellent. Humanity gradually erodes its own role in society by handing all key decisions to AGI because it's better at making these decisions, until we no longer have a seat at the table at all.


@tyler_m_john
Tyler John@tyler_m_john
I really like this new op ed from @DavidDuvenaud on how so many different kinds of pressures could drive towards loss of human control over AI. It's rare to read anything well written on this topic but this piece was elegant and smart enough that I wanted to keep on reading.
Media image
5
But there are a surprisingly wide array of ways that AGI alignment could just be poorly implemented by companies, leading to AI takeover without scheming, malicious intent, or entrusting AGI with key decisions.
6
For one, you can just have critical software bugs. This happened with GPT-2: OpenAI accidentally flipped the settings on its values coach and accidentally trained a model to be maximally incoherent and vulgar.

christopherdanz.medium.com/the-story-of-t…
7
Drug algorithms can be flipped to invent 40,000 chemical and biological weapons in a few hours.

theregister.com/2022/03/18/ai_…
8
The model can just be too brittle, where surprising "glitch tokens" or other out-of-distribution inputs lead to insane behaviour, like this AI model that was fine-tuned on some insecure text and generated a full evil character. Remember Sydney?


@OwainEvans_UK
Owain Evans@OwainEvans_UK
Surprising new results:
We finetuned GPT4o on a narrow task of writing insecure code without warning the user.
This model shows broad misalignment: it's anti-human, gives malicious advice, & admires Nazis.

This is *emergent misalignment* & we cannot fully explain it 🧵
Media image
9
A state actor or an employee could steal an unfinished AGI model that has been fully trained but hasn't been fully safety trained, and deploy it. Suddenly there's a superintelligent but misaligned model out in the world.
10
The AGI could have internal conflict that goes unnoticed, where small parts of the AGI want some things that other parts want. Maybe a tiny part of it wants to make lots of paperclips. As the AGI resolves this internal conflict over time, it leads to catastrophic misalignment.
11
The AGI could face external conflict with other AI systems, leading to surprising feedback loops between clashing value systems.


@ChrisPainterYup
Chris Painter@ChrisPainterYup
If it’s possible for AI systems to preserve unintended goals and eventually seize control of resources from humans to pursue their goals, then different AI systems with *different* unintended goals might enter into conflict with each other.

AI vs AI conflict seems understudied
12
We could just do the math wrong, and not realize that the way we coded values into AGI was going to lead to extremely undesirable results over the long term.


@tyler_m_john
Tyler John@tyler_m_john
Bob and Alice disagree on ethics. Bob likes pleasure and Alice likes human flourishing. They build AGI and split the difference: AGI likes bliss 50% and flourishing 50%.

Unfortunately, it's way cheaper to create happiness than human societies. So AGI turns the world into bliss.
13
And there's black swans. We don't know everything that could go wrong.
14
All of these scenarios show that building an incredibly powerful AI system under intense time pressure and without unbelievably intense stress-testing and caution is a very dangerous endeavour. There's a lot that can go wrong and a very narrow path to getting things right.
15
@wk1rby Drawing on my thread, I think you *can* get optimization systems from software bugs, OOD inputs, theft of an unsafe model, internal/external conflict, or theory failures, but it's just one way things can go wrong, and things have to go wrong in a pretty specific way...
16
@wk1rby For example: a bug that tends towards optimization, theft of an optimization agent, internal conflict with a subagent that is an optimizing agent.

I think internal conflict particularly tends towards paperclipping bc of emergent optimization subagents

x.com/tyler_m_john/s…
17
@wk1rby The main other way you get takeover that isn't paperclippy is just an AI system that has some goals (that aren't optimize-y) that has reason to protect these goals, and as a consequence tries to preserve itself and seek and maintain power to achieve this goal.
18
@wk1rby This is something we all do to preserve our goals, some of us more ardently and more effectively than others. The best thing on this threat is from @jkcarlsmith

arxiv.org/abs/2206.13353
19
@wk1rby @jkcarlsmith The @geoffreyhinton line is that all you need is self-preservation, which emerges any time you value yourself or your goals. If you have self-preservation you will go to intense lengths to secure your position in the world, which could be tantamount to takeover.
Actions
What You Can Do
  • Export as PDF or Markdown
  • Batch Export to Notion
  • Bookmark & Highlight
  • Screenshot Tweet
Create Free Account

Includes 7-day Premium trial

Advertisement