Announcing ARC-AGI-3 - A benchmark that tests if AI can explore, learn, and adapt in unfamiliar situations. Humans score 100%. Frontier AI scores 0.26%.

2026-03-27 04:44 UTC

The ARC Prize organization designs benchmarks which are specifically crafted to demonstrate tasks that humans complete easily, but are difficult for AIs like LLMs, “Reasoning” models, and Agentic frameworks. ARC-AGI-3 is the first fully interactive benchmark in the ARC-AGI series. ARC-AGI-3 represents hundreds of original turn-based environments, each handcrafted by a team of human game designers. There are no instructions, no rules, and no stated goals. To succeed, an AI agent must explore each environment on its own, figure out how it works, discover what winning looks like, and carry what it learns forward across increasingly difficult levels. Previous ARC-AGI benchmarks predicted and tracked major AI breakthroughs, from reasoning models to coding agents. ARC-AGI-3 points to what’s next: the gap between AI that can follow instructions and AI that can genuinely explore, learn, and adapt in unfamiliar situations. You can try the tasks yourself here: arcprize.org/arc-agi/3 Here is the current leaderboard for ARC-AGI 3, using state of the art models OpenAI GPT-5.4 High - 0.3% success rate at $5.2K Google Gemini 3.1 Pro - 0.2% success rate at $2.2K Anthropic Opus 4.6 Max - 0.2% success rate at $8.9K xAI Grok 4.20 Reasoning - 0.0% success rate $3.8K. (Logarithmic cost on the horizontal axis. Note that the vertical scale goes from 0% to 3% in this graph. If human scores were included, they would be at 100%, at the cost of approximately $250.) arcprize.org/leaderboard Technical report: arcprize.org/…/ARC_AGI_3_Technical_Report.pdf In order for an environment to be included in ARC-AGI-3, it needs to pass the minimum “easy for humans” threshold. Each environment was attempted by 10 people. Only environments that could be fully solved by at least two human participants (independently) were considered for inclusion in the public, semi-private and fully-private sets. Many environments were solved by six or more people. As a reminder, an environment is considered solved only if the test taker was able to complete all levels, upon seeing the environment for the very first time. As such, all ARC-AGI-3 environments are verified to be 100% solvable by humans with no prior task-specific training

Replies (27)

@Multiplexer@discuss.tchncs.de 2026-03-27 05:32

Link to the recent Al Explained video mainly covering ARC-AGI-3: www.youtube.com/watch?v=s4tptozUJ8Y

Open ##906760
@lath@lemmy.world 2026-03-27 05:48

Biased study. Take any average person off the streets and shove this thing in their face. That 100% notion will go down fast.

Open ##906932
@ExLisper@lemmy.curiana.net 2026-03-27 08:38

Can’t wait for this to be the new captcha.

Open ##909784
@HaunchesTV@feddit.uk 2026-03-27 08:47

Grok Reasoning: 0% Hilarious

Open ##909885
@Bubbaonthebeach@lemmy.ca 2026-03-28 03:44

I tend to be anti-AI because it doesn’t seem to me to be anything other than a super fast regurgitator of data. If a database can be searched for an answer, AI can do that faster than a human. However it doesn’t to seem to be able to take some portion of that database, understand it, and then use that information to solve a novel problem.

Open ##915496
@RustyShackleford@piefed.social 2026-03-27 05:53

As a psychiatrist, I have a theory about what’s missing in AI. First, it lacks childhood dependency and attachments. Second, it struggles to overcome repeated pain and suffering. Third, it lacks regular eating and restroom breaks. Fourth, it struggles to accept loss in everyday situations. Finally, it lacks the concept of our inevitable death. Without these nagging memories and concepts, machines will simply revert to the simpler concepts we use them for in our recent times, such as stealing cryptocurrency. After all, we live in a world run by capitalism, so it’s only logical. ¯\\_(ツ)_/¯

Open ##1302696
@GreatBlueHeron@lemmy.ca 2026-03-27 11:14

It's fun to point at the crappy performance of current technology. But all I can think about is the amount of power and hardware the AI bros are going to burn through trying to improve their results.

Open ##1302697
@SaraTonin@lemmy.world 2026-03-27 19:21

Tell me again how AGI is just around the corner, Sam

Open ##1302698
@HertzDentalBar@lemmy.blahaj.zone 2026-03-27 18:11

It's almost as if a chatbot isn't actually thinking.

Open ##1302699
@mechoman444@lemmy.world 2026-03-28 03:25

I know lemmy's very anti-ai but this is really fascinating stuff.

Open ##1302700
@fox2263@lemmy.world 2026-03-28 07:09

I can’t see AI actually being intelligent until they no longer need to send a built up prompt of guides and skills and the chat history on every submission. It’s no different from Alexa 15 years ago with skills. Just a better protocol and interface and ability to parse the current user prompt. In my opinion of course.

Open ##1302701
@Tetragrade@leminal.space 2026-03-27 15:29

This replay is the funniest shit lmao. Keep building that bridge Claude. https://arcprize.org/replay/0964128b-a2f5-4c5b-886e-497d893f429d Interesting that it seems to be perceiving the environment mostly accurately, and is just completely wrong about the purpose of all the game objects.

Open ##1302702
@UnrepentantAlgebra@lemmy.world 2026-03-27 12:52

>If human scores were included, they would be at 100%, at the cost of approximately $250 Wait, why did it cost real humans $250 to pass the test?

Open ##1302704
@arcine@jlai.lu 2026-03-27 22:34

Try spelling things phonetically (example: faux net tick alley), that's one of my benchmarks that AI fails almost every time. If the input is at all long, or purposefully includes a lot of words about a specific unrelated theme to the coded message, it's impossible.

Open ##1302705
@sunbeam60@feddit.uk 2026-03-27 20:22

Ii can thoroughly recommend “A Brief History of Intelligence” (by Max Bennett), which explains how intelligence has taken steps through evolution, what those steps were etc. Spatial intelligence requires spatial understanding and it’s not something that can be solved through a large language model, IMHO. I’m excited to see how these are solved. And I’m terrified to see how these will be solved.

Open ##1302706
@General_Effort@lemmy.world 2026-03-27 13:10

>ARC-AGI-3 What happened to ARC-AGI-1 and -2?

Open ##1302707
@communist@lemmy.frozeninferno.xyz 2026-03-28 22:01

They get 85% on the last benchmark, this one was specifically designed to stump them, when the last one came out everyone said the same things as this go around. will anyone be retracting their statements when they get to 85% on this one?

Open ##1302711
@SuspciousCarrot78@lemmy.world 2026-03-27 17:33

"...specifically crafted to demonstrate tasks that humans complete easily" Motherfucker, I can't work out Minesweeper. I got zero fucking chance with your mystery box bloop game.

Open ##1302712
@Kissaki@feddit.org 2026-03-27 19:56

"Leaderboard" where rank one scores 0.3% success lol

Open ##1302715
@ayyy@sh.itjust.works 2026-03-27 17:24

The humans literally didn’t score 100% though. Why lie?

Open ##1302716
@Diurnambule@jlai.lu 2026-03-27 17:54

Boring game...

Open ##1302717
@Sam_Bass@lemmy.world 2026-03-27 16:34

AI code is prewritten and is unable to edit that. Humans edit their "code" every second

Open ##1302718
@WorldsDumbestMan@lemmy.today 2026-03-27 15:07

I'm not sure such a general term is factual. I doubt I can adapt 100%

Open ##1302719
@mindbleach@sh.itjust.works 2026-03-28 13:31

Optimistically - this is now something backpropagation can reward or punish, to make models of any design act smarter. Pessimistically - when a metric becomes a goal it ceases to be a useful metric.

Open ##1302720
@tatterdemalion@programming.dev 2026-03-27 12:29

LLMs might suck at this game but I'm pretty sure Deepmind's deep reinforcement learning AI could solve these easily. EDIT: I know you guys hate AI around here, but you need to at least be aware of what the technology is capable of. From 11 years ago: https://youtu.be/V1eYniJ0Rnk

Open ##1302721
@mechoman444@lemmy.world 2026-03-29 02:46

The analogy is terrible and is not at all, once again, what llms do. This is an objective fact I have provided evidence to support this. How are you saying the analogy is good?

Open ##1618962
@mechoman444@lemmy.world 2026-03-29 05:19

I fully understand the analogy being presented. It is a poor analogy and fundamentally incorrect because that is not how LLMs function. They do not “read back Wikipedia pages,” which is a complete misunderstanding of the technology, not a minor lack of precision. I am not disputing that it is an analogy, nor am I claiming that exact precision is necessary to analyze it. The point remains: the analogy fails. What is curious is how people focus on my tone, saying I am aggressive or should be more precise, rather than engaging with the substance of my argument. So far, no one has directly refuted my points. This suggests that many responding are simply following the anti-AI bandwagon without understanding the technology, which is both reductive and disappointing.

Open ##1618963