OpenAI o3 scored 91.5% on public ARC-AGI test despite taking 57M...
Deedy@deedydas
56 views
Dec 22, 2024
~1 min read
Advertisement
1
OpenAI o3 scored 91.5% on public ARC-AGI test despite taking 57M tokens and ~$5000/task. The problems are fairly simple!
I made an app with Claude to visualize them. Strangely, Sonnet can even identify the pattern sometimes.
Let's look at the 34 problems o3 got wrong...
1/13
I made an app with Claude to visualize them. Strangely, Sonnet can even identify the pattern sometimes.
Let's look at the 34 problems o3 got wrong...
1/13
2
The spiral with a margin of one.
2/13
2/13
3
The connected rectangle fillers.
3/13
3/13
4
The antidote shield padding.
4/13
4/13
5
The dumb traveller with breadcrumbs.
5/13
5/13
6
Rectangle sliding.
6/13
6/13
7
Copy paste my pattern.
7/13
7/13
8
The semi-symmetric eraser.
8/13
8/13
9
The legend that guides overlapping squares.
9/13
9/13
10
Count the height of overlapping rectangles.
10/13
10/13
11
Connected rectangle outlines.
11/13
11/13
12
Complete the triangle.
12/13
12/13
13
The cropped spiral.
13/13
13/13
14
Another instance of the semi-symmetric eraser.
15
Artifact for visualization: claude.site/artifacts/af7f…
Public benchmark data: github.com/fchollet/ARC-A…
o3 results and attempts: github.com/arcprizeorg/mo…
@fchollet @GregKamradt
Public benchmark data: github.com/fchollet/ARC-A…
o3 results and attempts: github.com/arcprizeorg/mo…
@fchollet @GregKamradt
16
Examples covered: da515329, f9d67f8b, 891232d6, 52fd389e, c6e1b8da, 09c534e7, ac0c5833, 47996f11, b457fec5, b7999b51, b9630600, 896d5239, 40f6cd08, 8b28cd80