Humans Near a Perfect Score. The Best AI Still Gets 39%

Nathan Labenz opens The Cognitive Revolution with a contest designed to yank AI discourse off the language-model treadmill. Mike Knoop, co-founder of Zapier, has co-created the ARC Prize: a $1 million public competition for more sample-efficient, generalizable architectures, including a $500,000 grand prize for a system that solves the ARC-AGI benchmark at a human-like level under strict compute and time limits. ARC is the Abstraction and Reasoning Corpus, built by François Chollet in 2019 as a test of general intelligence. Each item shows input-output pairs of two-dimensional grids that demonstrate a transformation, plus a final input to solve. The solver has to infer a rule that is different in every puzzle, then apply it. The test set stays private so systems cannot memorize answers. Samples live at arcprize.org.

Labenz is not fully sold on the gloom. He has used large language models intensively for three years and finds the progress in reasoning unmistakable, even if incomplete. He still thinks it matters that GPT-4o and Claude 3.5 Sonnet struggle with ARC-style puzzles. When he solves them himself, a subconscious, deeper-than-language intuition does most of the work: he stares, then a eureka moment arrives, and the rest is bookkeeping. Current systems are nowhere near that kind of insight, which is also what you would need from an AI scientist that proposes hypotheses worth testing in the real world. A technique that solved ARC inside the contest rules would, he says, be a notable step toward AGI.

The same efficiency that makes the contest interesting makes him nervous. Knoop later suggests that nature's tight energy budget may be why intelligence exists at all: organisms that made good decisions from limited local evidence survived. Labenz wonders whether a breakthrough architecture that solves ARC for about a cent of compute would have room for the kind of values understanding Claude shows today. Early AI-safety arguments worried about small, highly capable systems — the paperclip maximizer — that solve problems without knowing when to stop. After the interview he says he tentatively hopes a hybrid wins: a language-model component that understands human values, nested around more algorithmic search. That ensemble would not be eligible for the prize. If it is fast and cheap enough to compete with humans, he doubts the brute-force objection will matter in daily life.

Knoop co-founded Zapier and ran all of product engineering until about mid-2022. The chain-of-thought paper that January knocked him off that path. He handed the job back and returned as an individual-contributor AI researcher, because he needed to know, for Zapier's business and as a human being, whether the field was on track for AGI. He had first heard Chollet during COVID, on a podcast with Lex Fridman, talking about *On the Measure of Intelligence* and ARC. As he started shipping AI inside Zapier he got obsessed with benchmarks. They define and guide system quality, and almost every public AI benchmark was saturating to human level — faster and faster as language models scaled.

ARC was doing the opposite. Progress had been decelerating for four years since 2019. That, Knoop decided, was not an accident. Four weeks into the enlarged prize, nobody had come forward with a better AGI benchmark, or even another AGI benchmark. ARC, he argues, is still perhaps the only test in the world singularly designed to measure AGI and separate it from ordinary AI, and the fact that it remains unbeaten is why he pitched Chollet to blow up the prize pool. Labenz offers a smaller parallel from his own work: aesthetic evaluation of images used to look similarly stuck, then multimodal models largely solved it. Knoop grants that other hard boards exist — he mentions SWE-bench's very low state of the art — but says what is unique about ARC is that it was built from a definition first.

Knoop sees two public schools of thought. One is we'll-know-it-when-we-see-it: AGI is undefinable, we are already on track, stop worrying. The other, popularized by OpenAI, says AGI is a system that outperforms humans at the majority of economically useful work — language that is literally written into OpenAI's agreement with Microsoft, because hitting it changes IP ownership of future systems. He does not think that is a bad goal. He thinks narrow AI can already accomplish it, which is why it is a poor definition of generality. A system that wins at poker, Go, chess, bits of self-driving, or Diplomacy looks impressive until you notice that each win required researchers to start from zero: new search, new data, new scale. Sit a person down with a new card game and they are at human proficiency in a few hours of experience.

The definition he wants is the one Chollet wrote in 2019: AGI is a system that can efficiently acquire skill and apply it. Skill-acquisition efficiency is the hallmark. Narrow skill can be memorization. Rapid transfer to tasks you have never done is generality, and that facet, Knoop says, has stalled for three or four years while the industry got extremely good at targeting narrow systems. No one knows how to build a machine that acquires a never-seen domain at human efficiency. That insight is what ARC sits on. It is not a random hard quiz. Four years of unbeaten, slowing scores plus a conceptual frame is why he believes progress toward the generality of these systems has actually stalled.

The tasks are a minimal reproduction of general intelligence. Every one is supposed to be novel for the solver and for any system built to beat it. There is a public set for training and design, and a private set the competition actually scores. Very few humans have seen the private puzzles, which is why an 85% result — the grand-prize bar — would be evidence of generalization rather than of having trained on answers or on other people's solver code. You do not need world knowledge or language. The items rest on about seven or eight core knowledge priors children pick up very early: objectness, goal-directedness, symmetry, rotation, masking. That is why adults find ARC straightforward and why, so far, no AI system does.

The files themselves are JSON: two-dimensional arrays of digits from 0 to 9, a few demonstration input-output pairs, then a test input. The colorful grids on the website are just viewers. Colors are arbitrary symbols; you could swap the palette and every puzzle would still be solvable. Zero-as-black might be a special case in some items, but humans handle that. The job is to emit the right output grid, whether you get there by writing a program or by any other means. Labenz notes that six examples on the ARC Prize homepage are enough to feel the "aha."

The 2024 contest uses the exact same private set Chollet created in 2019, reused in 2020, 2022, and 2023. Keeping a test set dark for five years looks, in hindsight, like foresight; Scale's SEAL benchmark is now doing something similar. ARC is not perfect. Chollet did not expect it to last this long, and he built it with the resources he had. After the 2020 Kaggle round, analysis suggested the first-place individual result was only about 20%, and AI models were barely a technique. Ensemble every submitted solver and you could brute-force 40 to 50% of the puzzles. Private-set state of the art on Kaggle at taping was still 39%. Model-based methods are, in some respects, only now matching what search already could do. Getting to 85% will require something beyond brute force, because combinatorial explosion and minimum description length take over.

They will not change the benchmark mid-contest. Between annual windows they want to stair-step ARC alongside system strength: more novelty, less brute-forceable residue, a larger set than 100 private items, which is statistically thin. Today every task is still hand-designed and hand-verified as human-solvable, with some later crowdsourcing. Knoop expects that once weak AGI-like systems exist, those systems will help design harder tests — a takeoff sign Labenz jokes about. Meanwhile the landscape splits into private in-house evals (Zapier has its own), LMSYS-style Elo battles that borrow human aesthetic judgment, and a coming wave of unseen, private tasks in the spirit of ARC and Scale's SEAL, one-shot so overfitting is harder.

Four weeks in, Ryan Greenblatt of Redwood Research sat atop the public leaderboard at about 42%. His method feeds puzzle images into GPT-4o, samples thousands of Python programs, keeps the ones that reconstruct the demonstrations, and applies those to the test input — on the order of 2,000 reasoning traces per puzzle, versus tens or hundreds of thousands in 2020 brute force. Skeptics say GPT-4o saw ARC on GitHub. Knoop does not buy direct leakage of the public answers as the main story. For years people have posted DSLs and per-puzzle solver code; that code is what is in the pretraining mix. When a language model samples programs, it is drawing on how humans solved similar-looking items. The approach still matters: scores agree with a new semi-private set, and using the model as a perception engine to guide program search is genuinely new.

The grand prize is 85% on 100 hand-verified private puzzles, submitted on Kaggle by November 10, 2024, with the solution committed to the public domain. Pretrained models and licensed software are allowed. The internet is not, which knocks out GPT-4o APIs. Compute is a P100 for 12 hours of runtime. Labenz prices a used P100 around $300 on Amazon at 250 watts, which amortizes to about a dollar of compute for the whole run, or 7.2 minutes and roughly one cent per puzzle; on-demand cloud might be $20 or $30 for the twelve hours. An 8B Llama-class model should fit in the card's memory; not much bigger. To collect any prize money, including progress prizes, you must publish. Knoop expects multiple years, with each contest dumping open approaches so the community can re-baseline.

Private-set state of the art, 39%, belongs to Jack Cole's team. They use a Salesforce CodeT5-class model of about 220 million parameters and do test-time fine-tuning: at runtime they generate new puzzles in response to the item they are shown, fine-tune, then prompt. The model is small because that is what fits the budget. That result is why Knoop, forced to bet, says ARC will probably fall to a 7B model and 10,000 lines of code. Training is unlimited; only inference-time compute is capped. Last year Kaggle donated an even weaker GPU for three or four hours; this year they roughly doubled or tripled it. The cap exists because Chollet's definition includes efficiency. If efficiency does not count, general intelligence collapses into brute-force search. Humans do not enumerate thousands of transforms. Perception and experience shrink the set to three or five candidates, then they check.

The public leaderboard is a separate curiosity board with no prize, internet allowed, and a $10,000 spend cap instead of a P100. Labenz estimates about $100 per puzzle there versus one cent on the private set — a 10,000-fold gap. State of the art on both boards is only a couple of points apart, and both camps are maxing out runtime or dollars. Existence proofs then waterline: once someone shows a score with more compute, others hunt for the same idea inside the contest limits. Knoop has seen the four-minute-mile pattern in the 2020 ARC Kaggle, in speedrunning, and now here. The public board is also the on-ramp. High scores ship code; copy Greenblatt's notebook, sample ten random puzzles instead of 500, and a trial run is $10 to $20.

The prize is also a reaction to the market. Because language models have some economic utility, labs have over-rotated. Knoop's rough 2023 count: about $20 billion into language-model startups, and maybe a couple of hundred million into people working on new architectures and learning algorithms — literally on the order of 20 to 1. Commercial value also closed the papers. OpenAI did not share how GPT-4 worked; Gemini did not share how the million-token context worked. He expects more of that. It is the opposite of the path that produced the stack everyone is using. Ilya Sutskever's sequence-to-sequence paper at Google in 2014 was open. Dzmitry Bahdanau's attention mechanism was open. Vaswani, Shazeer, and colleagues published *Attention Is All You Need* in 2017. Alec Radford and Ilya at OpenAI turned that into GPT-2. Today's progress, Knoop says, exists because of open sharing, and 2024 is not that world. ARC Prize is a small counterweight: new ideas, in the public domain.

Eligibility is checked at the end, including country restrictions on Kaggle payouts. Top scores must commit to publishing into the public domain; the organizers will act as clearing house when the money goes out. Labenz asks about the risk that someone scores high, then auctions themselves to a lab or a startup instead. Knoop's answer is the waterline. They will keep running the prize until a solution is in the public domain. He has seen Twitter jokes offering a $10 million starting salary to anyone who has a solver and will not show the code. He actually likes that. It means people are pricing new ideas against the "scale is all you need" story. The only way open AGI progress happens now, he thinks, is if more researchers can be convinced to work on new ideas again.

Labenz offers his own definition, shaped by Zapier-like workflows: intelligence is the ability to succeed on a task where we do not have an explicit algorithm. He asked Claude 3.5 Sonnet to write explicit code for MNIST-style digit recognition and got about 14%, barely above chance. There is still no good non-learning algorithm for MNIST, and humans and trained models both do it easily. Knoop says the spirit matches Chollet, but Labenz's version is less demanding: it still leaves a back door for brute-force search over all possible algorithms. Efficiency, Knoop argues, may be the gradient evolution used. Early organisms that randomly explored burned energy and hit predators. Making smarter local decisions toward food is what survived. Contest rules cap only inference. Unlimited pretraining is allowed, the same way a human's 7.2 minutes sit on evolutionary history plus a childhood. Core knowledge arriving at age one looks like architectural predisposition, which is itself an outsourced search across ancestral lifetimes.

That existence proof — general intelligence in every listener's head, architecture unknown — is why Knoop wants a broader architecture search. A decade into deep learning he counts maybe seven to nine mainline designs: CNNs, RNNs, transformers, xLSTM, RWKV, state-space models. Run out of new architectures in 2024 and you would be surprised in 2044. Each has different efficiency and inference-speed properties; none yet lets generality emerge as strongly as brains do. Labenz then introspects four steps on an ARC item. First, visual-spatial detection of salient features — a cross jumping off the screen. That is objectness, and Knoop calls it perception, a deep-learning strength, not the hard part, even if GPT-4o and Claude still look a bit blurry on the grids. Second, guessing the rule: an intuition that bubbles up from nowhere — fill the holes with yellow — which Labenz cannot explain and which he identifies with AI for science, the ability to propose hypotheses worth the wet-lab cost. Protein design is the early version of that.

Today's best language-model attacks brute-force thousands of rules and check them with deterministic code. Humans use perception, grounded in core priors, to shortcut to a handful. The near-term path Knoop likes is a deep-learning perception net steering a program-synthesis engine, with ordinary code as the checker. Nobody sat down and designed a synthesizer into the brain; it emerged. So there are two schools. Search new architectures and scales until synthesis emerges, which is inefficient but more likely to find a global solution we already know exists. Or build the engine directly from our intuition of the parts, which is faster and may miss something. Frontier labs, Labenz notes, would say scale will emit few-shot learning, the GPT-3 trick. Knoop's reply: bigger models need fewer prompt examples because more programs were stuffed into pretraining, but they are still doing direct inference off memorized training data. They do not invent programs with no bearing in the corpus. That is why he suspects current architectures top out around 50 or 60% and will not take the grand prize.

Labenz takes the other side of the scaling bet and cites two papers. In 2022, a U.S.–China collaboration encoded gradient descent in matrix form, then found the same pattern in normally pretrained weights — evidence, to him, that few-shot prompting is implementing a kind of runtime learning, not mere memorization. Knoop clarifies: language models clearly generalize, which is why they have economic utility, and they are more general than anything before the 2017 transformer. What has not progressed, he says, is the underlying generalization architecture. All current language models still use that same one. The second paper, *On the Paradox of Learning to Reason*, trained a BERT on 500 synthetic deductive rules and got 100% in-distribution, then watched accuracy fall below 50% as hop count rose on a shifted 500. Researchers then hand-coded BERT weights, abusing broadcasting in the architecture, and hit 100% on both distributions. Backprop never found those weights. The general intelligence in that experiment lived in the researchers.

One reading, which Knoop finds interesting, is that the architecture may be sufficient and the learning algorithm is wrong. Chollet has also said that if you generate every possible ARC puzzle and train on them, you can probably prompt your way through, a known deficiency. Unlimited pretraining plus a data-center brute-force search can beat the set today; the compute cap and no-internet rule are there to raise confidence that a score is real. For v2 they want to study which tasks have been least susceptible to brute force across four years, grow the set — 100 private items is small — and add more of that high-novelty residue. Labenz finishes his four-step map: writing the program feels more algorithmic even without an explicit translation procedure; checking against demonstrations is mechanical; debugging a miss is the last loop. Greenblatt's write-up says the coding is pretty good but not awesome, and a revision step helps, while perception and rule-guessing still force a huge sample count.

Humans do something else the happy path leaves out. The public set has had stray wrong pixels and swapped demonstrations. A few puzzles have intentional ambiguity that needs the two allowed submissions. People hold multiple candidate rules, notice they have insufficient information, make contact with reality, and update. Today's AI systems basically do not. The organizers have chosen to remove mistakes rather than turn robustness into the test, but Knoop thinks an ARC-like set full of intentional errors would be a fascinating adjacent benchmark. On his own, before launching the prize, he ran prior-year open solutions and frontier models, landed around 20% without much iteration, and noted that some 300 teams in the 2023 Lab42-era contest had not done much better. His side project is more indirect: skip backpropagation, search for an architecture from which program synthesis emerges, and treat ARC as feedback for that search rather than as the goal.

Neural architecture search is an old idea, possibly the largest compute sink in the field before the transformer, and it has discovered almost nothing interesting. Academic labs, short on compute, took shortcuts that collapsed NAS into hyperparameter tuning — layer counts, learning rates. A true relaxed search, in Knoop's head, looks like a dense net where you prune connections. Ternary and binary weight results this year excite him because deciding whether two nodes connect, zero or one, is defining an architecture, not just optimizing weights. The language-model boom drove flops per dollar down far enough that a global architecture search for AGI primitives might finally be affordable, which it was not three or four years ago. It is a slower bet than a direct ARC solver, and the takeoff implications, Labenz notes, are striking: automated architecture finding, and the world's oyster.

Asked what an AGI world looks like if architecture search works, Knoop does not reach for the singularity. Near term, he says, it will be very boring: fixing exacting accuracy at the application layer. After a year and a half of deploying AI at Zapier, the binding constraint is user trust — not privacy theater, but whether people will run a system with no oversight. They either keep a human in the loop, which re-caps the system at human speed, or they refuse to deploy because the risk is too high. A real incident: an AI system chose which Slack channel to post to and sent user content into a channel shared with outside partners. The fix was a deterministic override. The model may write the message body; it must never guess the channel. If you solve ARC, you have a program that generalizes from relatively arbitrary core-knowledge priors and, with exacting accuracy, solves tasks it has never seen. That is a developer toolkit for reliability, not a god.

He draws concentric rings of use cases. The inner ring is low-regret personal productivity: summarize my inbox. Outer rings — customers, production — have catastrophic downside, and willingness to fully automate them is near zero today. Labenz finds it counterintuitive that out-of-domain generalization would be what unlocks in-domain reliability; his own habit when he wants a Zap to behave is to narrow the model with examples and chain-of-thought. Knoop's answer is that the priors are arbitrary. The nearest-form ARC solution is a deep-learning-guided DSL generator plus a deep-learning-guided synthesizer, not tied to Chollet's original priors. Zapier's AI bots already let users paste API keys and write rules in English — every hour, look at my inbox, if someone important emailed, summarize to Slack. Failures happen at each function-calling step unless the training data is dense enough that the product of per-step accuracy is still usable.

Give that toolkit to programmers and line-of-business users and they stop writing wrappers, DSLs, and a priori control flow. They program by a couple of examples and the system generalizes over the runtime novelty that breaks today's bots — your old boss emails from a personal address. What you probably want in that moment is for the system to ask. Zapier has tried to teach bots to escalate when inputs diverge from past successes; the decision to intervene is itself unreliable, so mistrust returns. The only loop that has made bots worth paying for is handing users full control of the prompt. Write "summarize my boss to Slack," discover you never named the boss, add the work address, discover the personal Gmail, add the name. That local steering beats a single meta-prompt chained at 80% per step. The AGI-shaped version, Knoop wants, is a system trained on the core priors of what users keep rediscovering, so each customer does not have to walk the same rakes. Solving ARC is not robots becoming perfect. It is, at minimum, a transformer-scale new branch: exact generalization from arbitrary priors, with all the follow-on questions of limits, speed, and deployment still ahead.

Knoop leaves on a coin flip he likes either way. If somebody beats ARC in six months, fascinating; if nobody does, also fascinating. Labenz thinks people can get there on the public board with 100 times the compute; the limited P100 budget is the real breakthrough if it falls. For name-making entrants, cost is the barrier. Sample ten puzzles instead of all 500 (the $10,000 cap covers 400 public plus 100 verification). Prove the algorithm on cheap open models, then scale. After Knoop goes to lunch, Labenz stays on for a postscript of concrete pointers, with a tree chipper grinding in the neighborhood, for anyone who wants to take a swing.

First, DSPy, the framework for optimizing natural language programs. It beat prompt engineer Sander Schulhoff in a head-to-head, which is a strange generalization result if the system were only memorizing. Labenz's first experiment would be to run DSPy on Greenblatt's Redwood solution and see how much juice the optimizer adds. Second, state-space models on images with multi-way scans, from a prior Mamba episode: ARC grids are 2D, SSMs want sequences, so people patch and scan four ways, six ways, forward and back, then merge. Multi-resolution tokenization — one cell, a neighborhood, a zoomed-out whole — plus SSM efficiency on long state, feels like a different angle. Third, specialized small models for each of his four steps, maybe a small Phi for rule-guessing, because the human existence proof is modular: visual cortex, hippocampal indexing of the sort HippoRAG discusses, separate language. One model for guessing the rule and writing the code may be the wrong cut.

From DeepMind, AlphaGeometry mixes a language model that offers intuitions with a symbolic deduction engine that hammers them, a Thinking Fast and Slow loop that reached IMO-geometry gold-medal level after 100 million synthetic moves. That volume sits inside the letter of unlimited pretraining and a bit outside the spirit, if generating every puzzle is exactly what Chollet does not want. *Transformers Meet Neural Algorithmic Reasoners* couples the two with cross-attention: a graph module that groks classical algorithms beyond its training sizes, a transformer on the text, communication that is messy the way brain modules are. You would have to map ARC onto a graph form. FunSearch is the evolutionary cousin: language models write programs, a database keeps high scorers, a diversity trick fights local minima, and the loop set new records on notorious math problems. Finally Kolmogorov–Arnold Networks, KANs, from Ziming Liu and colleagues, invert the MLP: edges learn functions, nodes sum. Early, unscaled, aimed at AI for science and composable, interpretable formulas. Dynamically composing transformations is what an ARC program is. And because the private leaderboard already does test-time fine-tuning, he would not ignore grokking — modular addition took on the order of a million steps; whether a tiny net can grok one of these patterns in the time limit is unclear and worth a try.

The through-line in Labenz's list is the same as Knoop's near-term picture: let a neural system's intuition steer symbolic search, and do not expect one model to do every job. A hybrid that nested those solvers inside a values-aware language model would not win the prize. For transforming daily life, he does not think a functional general intelligence needs to satisfy our preferences about how intelligence ought to look. Creating a benchmark that has stood for more than five years is, he says, an awesome accomplishment, and putting real money on it is the part the rest of the industry should copy.