Part Ⅳ · The competition history
Three benchmarks, three competitions, one moving target. From a solo coder winning the 2020 Kaggle to o3's near-human leap, from the 2025 refinement loop stalling at 24% to the barely-begun agentic frontier — here are the teams and the scores, benchmark by benchmark, best-effort first.
Plate · The Climb & the Resets
One chart tells the whole story. ARC-AGI-1 crawled, then jumped to o3's 87.5% — so the ARC Prize rebuilt it as ARC-AGI-2, where the best score is 24%; then reinvented it as the interactive ARC-AGI-3, where the best agent barely clears 12%. Each colored line is a benchmark's frontier; each reset reopens the gap to the ~98–100% humans hold.
ARC-AGI-1 · 2020 → 2024
The original benchmark ran two great competitions — the 2020 Kaggle challenge and the 2024 ARC Prize — and crept from ~0% to ~5% until, in one season, test-time adaptation drove the private record from 33% to 55.5% and OpenAI's o3 vaulted to 87.5%. Here's the record, in order.
The first public competition, in 2020, was won by a solo entrant with a hand-built search and no learning at all. It set a template — DSL + brute-force + exact-match verifier — that held for three years while large language models stalled below 5%.
2024 was the break. The ARC Prize's first big season found that per-task adaptation, not scale, was the lever: test-time training and LLM-guided program synthesis pushed the private state of the art from ~33% to 55.5%. Then o3, spending enormous test-time compute on guided search, cleared the mid-80s — and forced the benchmark's redesign.
A hand-built DSL of ~142 grid functions, brute-force-composed into a DAG of pieces, keeping only programs that reproduce every training pair.
Why it mattersPure symbolic search, no learning — the blueprint every later method had to beat.
№07 In the Atlas →Sample ~8,000 GPT-4o Python programs per task, keep only those that replay every pair, majority-vote the survivors, then repair the near-misses.
Why it mattersShowed sample count is a real axis of accuracy — the ancestor of the 2025 refinement loop.
№10 In the Atlas →Fit a fresh LoRA adapter to each task's own demonstrations (with augmentation), a ~6× gain — matching average human when ensembled with program synthesis.
Why it mattersFormalized TTT for ARC; 2nd place in the 2024 Paper Award track.
№08 In the Atlas →TTT on an open 8B model, then score each candidate under every symmetry of the task and multiply the probabilities — only jointly-plausible answers survive. All offline at ~$0.02/task.
Why it mattersFirst place in the 2024 ARC Prize Kaggle competition.
№09 In the Atlas →A transductive deep-learning pipeline with full-parameter test-time fine-tuning and augment-infer-vote — the lineage that pioneered test-time adaptation on ARC.
Why it mattersHighest score of the season, but kept private — ineligible for prize money.
№21 In the Atlas →A genetic algorithm over Claude-written Python transforms: breed, score against the demos, keep the fittest, recombine across generations.
Why it mattersThe open, reproducible record on ARC-AGI-Pub — until o3 landed the same month.
№11 In the Atlas →At test time, search over many candidate chains-of-thought with a learned evaluator — the low-efficiency run spent ~172× the compute to reach 87.5%.
Why it mattersFirst human-competitive score — at tens to thousands of dollars per task. Not a competition entry; a frontier-lab result that triggered ARC-AGI-2.
№06 In the Atlas →ARC-AGI-2 · 2025
The 2025 ARC Prize was the first run against the harder, compositional ARC-AGI-2 — with efficiency as a first-class axis. ~1,455 teams entered; the top private score was just 24%, and the 85% grand prize went unclaimed. The winning pattern of the year was the refinement loop.
ARC-AGI-2 keeps the grid format but curates tasks that need multiple interacting rules, sequential steps, and in-context symbol definition — exactly what frontier reasoning models fail at. Every task is solved by ≥2 humans; models crater to the low tens of percent at best.
The defining move of 2025 was the refinement loop: propose a candidate, verify it against the training pairs, turn the error into feedback, and iterate. It came in three flavors — evolutionary program synthesis, application-layer harnesses wrapping a commercial model with a verifier, and weight-space refinement of tiny from-scratch networks. And because cost is now scored, the cheap winners matter as much as the strong ones.
Generate millions of synthetic look-alike tasks, then test-time-train a compact (~4B-parameter) model — reaching the top private score at about twenty cents a task.
Why it mattersCheap and small beat expensive and large — the whole point of ARC-AGI-2's cost axis.
№05 In the Atlas →Wrap a frontier model's chain-of-thought in a verifier that runs each candidate against the demos and feeds failures back — climbing from ~31% to ~54%, at ~$31 per task.
Why it mattersStrong, but ~150× the winner's cost per task — the accuracy-vs-efficiency tradeoff, made visible.
№05 In the Atlas →A ~7-million-parameter network, no pretraining, fits its weights per task using architectural recursion in place of search depth — ~45% on ARC-AGI-1 from scratch.
Why it mattersAstonishing on ARC-AGI-1, but drops to ~8% on ARC-AGI-2's compositional tasks — the honest measure of the gap.
№05 In the Atlas →The strongest general models, prompted directly — no ARC-specific scaffolding. They fit reasoning chains end-to-end and collapse on genuinely novel composition.
Why it mattersThe baseline that shows ARC-AGI-2 did its job: reopen a wide, real human-machine gap.
№02 In the Atlas →ARC-AGI-3 · 2025 preview
ARC-AGI-3 is the interactive benchmark: an agent dropped into a game with no instructions. It hasn't had a full competition yet — only a 30-day Preview Challenge in mid-2025 — so there are no "winners," just the best agents to date. The pattern is striking: everyone searches, no one truly models.
Because every step of an LLM agent is gated by a model call, it manages only ~4,000 of the ~96,000 allowed interactions — so cheap, deterministic explorers that spend the whole budget beat LLM agents outright. The top preview agents all explore the state space; none induces a reusable model of the dynamics or infers the goal.
Hand-built harnesses can solve all three public environments at near-human efficiency, but the same environment can go from 0% harness-free to ~97% with a harness — and none of it transfers to the private sets. That's why the official leaderboard reports only harness-free systems, where frontier AI sits below 1% against a 100% human baseline.
A CNN+RL model predicts which actions actually change the frame, then runs informed search over the tiny action space guided by those predictions.
Why it mattersThe top preview score — but it searches the action space rather than modeling the dynamics or the goal.
Builds a directed graph of observed frames and searches over it to cover the environment systematically.
Why it mattersRoughly half the top score — informed search over observed states, not modeling.
A training-free explorer: connected-component vision + a directed frame-hash graph, always heading toward the nearest untested action — spending the full step budget where LLM agents stall.
Why it mattersThe one preview agent with a published paper; beat the LLM baselines to 3rd on the private leaderboard.
№17 In the Atlas →Wrap a frontier model in per-environment scaffolding (Python-queryable history, or orchestrator+subagents) to solve all three public environments at ~human action counts.
Why it mattersSolves seen environments, fails unseen ones — up to 0% harness-free vs ~97% hand-built. Seen-env performance is not progress.
The strongest models, one action per turn from a fixed tool-free prompt, no ARC-specific scaffolding — the honest developer-aware-generalization baseline.
Why it mattersSub-1% versus 100% human: failing at exploration, modeling, goal inference, and planning at once.
ARC-AGI-3 has only had a 30-day preview — this is the frontier, not a final leaderboard. The full deep dive on every solution, the space of approaches, and the open questions lives in Part Ⅱ.
→ Enter Part Ⅱ · the ARC-AGI-3 deep diveThe pattern
A solo DSL + brute-force search wins the first Kaggle. No learning; the template holds for 3 years.
Per-task fine-tuning and augmentation-ensembling drive the private record from 33% to 55.5%.
Guided search over chains-of-thought clears 87.5% — near-human, but at thousands of dollars a task.
Propose → verify → feed back → iterate. Cheap small models beat costly big ones. Top score: 24%.
Interactive worlds. Everyone searches, no one models. Best agent so far: 12.58% vs 100% human.
Read the three competitions together and one shape appears: every time the machines catch up, the benchmark is redrawn to reopen the gap. Icecuber's search gave way to test-time training; test-time training gave way to o3's guided search; o3 forced ARC-AGI-2, where the refinement loop reigns; and ARC-AGI-2 begat the interactive ARC-AGI-3, where even the best agents barely clear 12%.
The techniques evolve, but the winning idea is remarkably stable: adapt to the task in front of you, and keep only what verifies. In 2024 that meant fine-tuning on the demos and voting across symmetries; in 2025 it meant iterating a candidate against an execution oracle. The open frontier — ARC-AGI-3 — is hard precisely because that clean verifier is gone: there are no demonstration pairs, no stated goal, only a world to probe.
→ How each of these actually works, in Part Ⅴ · Techniques