OpenAI o3 scored 91.5% on public ARC-AGI test despite taking 57M...

1

OpenAI o3 scored 91.5% on public ARC-AGI test despite taking 57M tokens and ~$5000/task. The problems are fairly simple!

I made an app with Claude to visualize them. Strangely, Sonnet can even identify the pattern sometimes.

Let's look at the 34 problems o3 got wrong...

1/13

2

The spiral with a margin of one.

2/13

3

The connected rectangle fillers.

3/13

4

The antidote shield padding.

4/13

5

The dumb traveller with breadcrumbs.

5/13

6

Rectangle sliding.

6/13

7

Copy paste my pattern.

7/13

8

The semi-symmetric eraser.

8/13

9

The legend that guides overlapping squares.

9/13

10

Count the height of overlapping rectangles.

10/13

11

Connected rectangle outlines.

11/13

12

Complete the triangle.

12/13

13

The cropped spiral.

13/13

14

Another instance of the semi-symmetric eraser.

15

Artifact for visualization: claude.site/artifacts/af7f…
Public benchmark data: github.com/fchollet/ARC-A…
o3 results and attempts: github.com/arcprizeorg/mo…

@fchollet @GregKamradt

                            16
                        
Examples covered: da515329, f9d67f8b, 891232d6, 52fd389e, c6e1b8da, 09c534e7, ac0c5833, 47996f11, b457fec5, b7999b51, b9630600, 896d5239, 40f6cd08, 8b28cd80

OpenAI o3 scored 91.5% on public ARC-AGI test despite taking 57M...

Actions

What You Can Do