Can AI really automate our jobs? New research tested seven LLMs:...
Aadit Sheth@aaditsh
20 views
Jan 04, 2025
~2 min read
Advertisement
1
Can AI really automate our jobs?
New research tested seven LLMs: Claude, GPT-4, Gemini, Amazon Nova, Meta Llama, and Alibaba Qwen.
Here's the jobs it could replace and the ones it could not 🧵:
New research tested seven LLMs: Claude, GPT-4, Gemini, Amazon Nova, Meta Llama, and Alibaba Qwen.
Here's the jobs it could replace and the ones it could not 🧵:
2
For the context, this is how it works:
Imagine creating a virtual company where AI assistants have to do real jobs, from writing code to managing projects.
That's exactly what these researchers at Carnegie Mellon University did!
Imagine creating a virtual company where AI assistants have to do real jobs, from writing code to managing projects.
That's exactly what these researchers at Carnegie Mellon University did!
3
They gave the AIs 175 different tasks you'd find in a typical workplace.
Things like:
• Writing and fixing code
• Managing team projects
• Dealing with HR stuff
• Handling company finances
Things like:
• Writing and fixing code
• Managing team projects
• Dealing with HR stuff
• Handling company finances
4
And the AIs had access to common work tools:
• Places to store code (GitLab)
• File sharing systems (OwnCloud)
• Task managers (Plane)
• Chat rooms to talk with AI coworkers (RocketChat)
• Places to store code (GitLab)
• File sharing systems (OwnCloud)
• Task managers (Plane)
• Chat rooms to talk with AI coworkers (RocketChat)
5
After a series of tests,
The final results came out...
The best-performing AI was Claude 3.5 Sonnet.
How'd it do?
Well, it could complete 24% of its tasks.
The final results came out...
The best-performing AI was Claude 3.5 Sonnet.
How'd it do?
Well, it could complete 24% of its tasks.
6
Each task was also pretty expensive, costing about $6.34 in computing power.
And the AI often needed around 30 attempts to figure things out.
Other models, like GPT-4o took fewer steps, but their success rate was quite low (~14 steps).
And the AI often needed around 30 attempts to figure things out.
Other models, like GPT-4o took fewer steps, but their success rate was quite low (~14 steps).
7
Is AI bad at everything?
No, it has some best-performing tasks too.
AI agents did best with technical stuff like coding.
But they struggled with things that required:
• Common sense
• Social skills
• Navigating complex websites
No, it has some best-performing tasks too.
AI agents did best with technical stuff like coding.
But they struggled with things that required:
• Common sense
• Social skills
• Navigating complex websites
8
Fun fact: Sometimes the AIs would make up solutions instead of solving the problem!
One AI agent even tried to "rename" a colleague on a chat platform instead of actually finding the right person!
One AI agent even tried to "rename" a colleague on a chat platform instead of actually finding the right person!
9
What does this tell us?
While AI is impressive, it's nowhere near ready to replace human workers.
They're more like helpful assistants who need lots of supervision.
The researchers suggest we need to work on:
• Teaching AIs better social skills
• Helping them handle unclear instructions
• Making them better at long-term planning
Think of current AI like an eager intern, they have potential but need lots of guidance and sometimes get things hilariously wrong!
This study shows we're still far from AI taking over the workplace.
For now, humans are very much needed to keep things running smoothly!
While AI is impressive, it's nowhere near ready to replace human workers.
They're more like helpful assistants who need lots of supervision.
The researchers suggest we need to work on:
• Teaching AIs better social skills
• Helping them handle unclear instructions
• Making them better at long-term planning
Think of current AI like an eager intern, they have potential but need lots of guidance and sometimes get things hilariously wrong!
This study shows we're still far from AI taking over the workplace.
For now, humans are very much needed to keep things running smoothly!
10
Thanks to @frankxu2004, @YufanSong98, @LiBoxuan91538, @gneubig, and @_rajmehta_ for carrying out this research and sharing it with the world.
Btw, I plan to share more of this stuff soon.
3 ways to get more (without fluff):
• Follow @aaditsh for AI trends
• Subscribe to my newsletter [link in the bio]
• RT the first tweet to educate your network
Btw, I plan to share more of this stuff soon.
3 ways to get more (without fluff):
• Follow @aaditsh for AI trends
• Subscribe to my newsletter [link in the bio]
• RT the first tweet to educate your network
Aadit Sheth@aaditsh
Can AI really automate our jobs?
New research tested seven LLMs: Claude, GPT-4, Gemini, Amazon Nova, Meta Llama, and Alibaba Qwen.
Here's the jobs it could replace and the ones it could not 🧵:
New research tested seven LLMs: Claude, GPT-4, Gemini, Amazon Nova, Meta Llama, and Alibaba Qwen.
Here's the jobs it could replace and the ones it could not 🧵:
11
If you liked this, join Prompts Daily — my newsletter with ~100,000 readers that tells you ONLY what you need to know about AI.
Link in my profile → @aaditsh
Link in my profile → @aaditsh