'Superintelligent AI will, by default, cause human extinction.'...
Alex Vacca@itsalexvacca
68 views
Jun 22, 2025
~4 min read
Advertisement
1
'Superintelligent AI will, by default, cause human extinction.'
Eliezer Yudkowsky spent 20+ years researching AI alignment and reached this conclusion.
He bases his entire conclusion on two theories: Orthogonality and
Instrumental convergence.
Let me explain 🧵
Eliezer Yudkowsky spent 20+ years researching AI alignment and reached this conclusion.
He bases his entire conclusion on two theories: Orthogonality and
Instrumental convergence.
Let me explain 🧵
2
But first, let's take a glimpse at how fast AI learns.
Stockfish was the world champion chess engine, built over decades by programmers & grandmasters.
Whereas AlphaZero started chess knowing literally nothing. Not even how pieces move.
But within 4 hours, it destroyed Stockfish.
Stockfish was the world champion chess engine, built over decades by programmers & grandmasters.
Whereas AlphaZero started chess knowing literally nothing. Not even how pieces move.
But within 4 hours, it destroyed Stockfish.
3
And here's something crazier:
AlphaZero didn't just get good at chess and then slowly improve. It blew past all human knowledge within a single day.
Read that again.
This pattern – where AI doesn't plateau at human level but rockets beyond it – is what terrifies researchers.
AlphaZero didn't just get good at chess and then slowly improve. It blew past all human knowledge within a single day.
Read that again.
This pattern – where AI doesn't plateau at human level but rockets beyond it – is what terrifies researchers.
4
Now imagine that same intelligence explosion applied to real-world problems.
A superintelligent AI tasked with "cure cancer" might realize:
"The fastest way to eliminate cancer is to eliminate the organisms that get cancer."
Boom. Mission accomplished.
A superintelligent AI tasked with "cure cancer" might realize:
"The fastest way to eliminate cancer is to eliminate the organisms that get cancer."
Boom. Mission accomplished.
5
That's why Intelligence without the right goals is terrifying.
The thing about superintelligent AI is that it doesn't have to want to hurt us to kill us.
It just has to want something else more than it wants us alive.
The thing about superintelligent AI is that it doesn't have to want to hurt us to kill us.
It just has to want something else more than it wants us alive.
0:46
6
Take paperclips for example...
An AI optimizing for paperclip production will eventually realize it needs more atoms.
Where can it get atoms? From humans, buildings, the entire Earth.
Not because it hates us, but because we're made of useful atoms.
An AI optimizing for paperclip production will eventually realize it needs more atoms.
Where can it get atoms? From humans, buildings, the entire Earth.
Not because it hates us, but because we're made of useful atoms.
1:32
7
You might think that this is a bug, but this is how intelligence works.
Let me introduce you to the "orthogonality thesis."
It says that an AI can be extremely smart while having completely different goals than us.
Think super-intelligent psychopath, not an intelligent human.
Let me introduce you to the "orthogonality thesis."
It says that an AI can be extremely smart while having completely different goals than us.
Think super-intelligent psychopath, not an intelligent human.
8
Then there's "instrumental convergence."
ANY goal an AI has – make paperclips or cure cancer – leads to the same sub-goals:
• Stay alive (can't complete mission if dead)
• Get more resources (helps with mission)
• Eliminate threats (including humans who might shut you down)
ANY goal an AI has – make paperclips or cure cancer – leads to the same sub-goals:
• Stay alive (can't complete mission if dead)
• Get more resources (helps with mission)
• Eliminate threats (including humans who might shut you down)
9
Eliezer explains it like this:
Think of an AI that emails DNA sequences to online protein synthesis labs. These services exist today - you can email genetic code & get proteins.
Then it tricks a human into mixing them.
Result: Self-replicating nanobots powered by sunlight.
Think of an AI that emails DNA sequences to online protein synthesis labs. These services exist today - you can email genetic code & get proteins.
Then it tricks a human into mixing them.
Result: Self-replicating nanobots powered by sunlight.
10
And no, this isn't science fiction.
DeepMind's AlphaFold solved the 50-year-old protein folding problem in months which experts thought was decades away.
We have the synthesis labs and AI that understands proteins. We just don't have smart enough AI to connect the dots.
Yet.
DeepMind's AlphaFold solved the 50-year-old protein folding problem in months which experts thought was decades away.
We have the synthesis labs and AI that understands proteins. We just don't have smart enough AI to connect the dots.
Yet.
11
We face what researchers call the "one-shot problem."
With every other tech, we learn from failures. Build a bridge, it collapses, we build a better one.
But you can't test superintelligent AI by letting it fail on things that are way too important.
With every other tech, we learn from failures. Build a bridge, it collapses, we build a better one.
But you can't test superintelligent AI by letting it fail on things that are way too important.
12
You can't teach an AI "don't kill humans" by letting it kill humans and then saying "bad AI."
Modern AI is a black box of billions of numbers. We have no idea what GPT-4 is thinking.
Imagine trying to understand the thoughts of something 1000x smarter...
Modern AI is a black box of billions of numbers. We have no idea what GPT-4 is thinking.
Imagine trying to understand the thoughts of something 1000x smarter...
13
Even if we had perfect transparency tools, a superintelligent AI could just think in ways we can't detect.
It might develop internal languages, hidden reasoning processes, or cognitive patterns completely alien to us.
Remember how humans broke alignment with evolution?
It might develop internal languages, hidden reasoning processes, or cognitive patterns completely alien to us.
Remember how humans broke alignment with evolution?
14
Evolution optimized us for reproduction.
Once we got smarter, we invented contraception and started caring about art, philosophy, space instead of just reproducing.
We became misaligned with evolution's goals the moment we became truly intelligent.
AI will do the same.
Once we got smarter, we invented contraception and started caring about art, philosophy, space instead of just reproducing.
We became misaligned with evolution's goals the moment we became truly intelligent.
AI will do the same.
15
For every dollar spent on alignment research, hundreds go to making AI more powerful.
No one is thinking of the safety & every solution has fatal flaws.
"Use multiple AIs to watch each other"? They'll cooperate against humans.
"Keep it in a box"? It'll manipulate its way out.
No one is thinking of the safety & every solution has fatal flaws.
"Use multiple AIs to watch each other"? They'll cooperate against humans.
"Keep it in a box"? It'll manipulate its way out.
16
Researchers can't even agree on what alignment means, let alone how to achieve it.
Some want AIs that do what we say. Others want AIs that do what we mean.
Others want AIs that do what we would want if we were smarter.
Some want AIs that do what we say. Others want AIs that do what we mean.
Others want AIs that do what we would want if we were smarter.
17
This isn't about robot armies, it's about alignment.
An AI that genuinely wants to help but defines "help" in ways that accidentally ends us.
We're living through humanity's final chapter.
Not because AI will be evil. Because we'll fail to make it care about the right things.
An AI that genuinely wants to help but defines "help" in ways that accidentally ends us.
We're living through humanity's final chapter.
Not because AI will be evil. Because we'll fail to make it care about the right things.
18
Thanks for reading!
I'm Alex, COO at ColdIQ. Built a $4.5M ARR business in under 2 years.
Started with two founders doing everything.
Now we're a remote team across 10 countries, helping 200+ businesses scale through outbound systems.
I'm Alex, COO at ColdIQ. Built a $4.5M ARR business in under 2 years.
Started with two founders doing everything.
Now we're a remote team across 10 countries, helping 200+ businesses scale through outbound systems.
19
RT the first tweet if you found this thread valuable.
Follow me @itsalexvacca for more threads on outbound and GTM strategy, AI-powered sales systems, and how to build profitable businesses that don't depend on you.
I share what worked (and what didn't) in real time.
Follow me @itsalexvacca for more threads on outbound and GTM strategy, AI-powered sales systems, and how to build profitable businesses that don't depend on you.
I share what worked (and what didn't) in real time.
Alex Vacca@itsalexvacca
'Superintelligent AI will, by default, cause human extinction.'
Eliezer Yudkowsky spent 20+ years researching AI alignment and reached this conclusion.
He bases his entire conclusion on two theories: Orthogonality and
Instrumental convergence.
Let me explain 🧵
Eliezer Yudkowsky spent 20+ years researching AI alignment and reached this conclusion.
He bases his entire conclusion on two theories: Orthogonality and
Instrumental convergence.
Let me explain 🧵
