The Best AI Scores 39% on This Test. Zapier’s Co-Founder Put $1 Million on Beating It

Sonya Huang and Pat Grady open Sequoia’s Training Data with Zapier co-founder Mike Knoop two weeks into the ARC Prize, the public contest he co-founded and is funding with François Chollet. The hosts frame ARC as a rare eval that tries to measure whether a machine can actually learn, rather than parrot patterns in its training data. Before they get to the prize, they want the Zapier story: an existing application company that got to AI early and then wove it through the product.

Zapier is a workflow automation platform with 6,000 integrations, from Salesforce to Gmail. Most customers would not call themselves programmers. That, Knoop says, is the point: non-technical people get software to do work for them, which is also what people want from AI. He has been “AI curious” since college and even walked the company through the GPT-3 paper at an all-hands. The jolt came in January 2022, when the chain-of-thought paper landed. He thought he had already priced in what language models could do. “Let’s think step by step” convinced him they were reasoning tools, not just chat engines, and he gave up running product engineering to go back to being an individual-contributor AI researcher alongside co-founder Brian.

The internal high-water mark is Zap Templates, the public recipes that show users what Zapier can do. Each one has to inspire a customer and map JSON fields from one app to another, so they were historically handmade. Contractors produced about 10 a day, and the backlog was on the order of a million. A partner-marketing teammate with a freelance-writing background built a stack of Zaps on OpenAI that, whenever a new integration launched, proposed templates, wrote the pitch, and did the field mapping. People moved from the do loop to the review loop: they now sit in a spreadsheet and mark outputs good or bad. Because generation is cheap, the team does not even fix the bad ones. They throw them away and generate another batch. Production is about 1,000 a day, a 100x jump.

Knoop’s advice for other companies is specific. Look at the top or bottom of the funnel, close to a conversion that matters, and find high-volume manual work. Ask whether you can get the human out of doing the task and into reviewing it. Review is still required at today’s reliability, he says, but the ROI can be enormous. At recording Zapier was about to hit a run rate of 10 million AI tasks a month. He would like to be corrected if someone has a bigger example, but he thinks that may make Zapier the largest automated AI platform in the world: systems doing work without a human in the loop, 10 million times a month.

Knoop stepped off the exec team in 2022 in part to answer a personal question: are we on a path to AGI, or not? Scaling was clearly happening. He wanted to know if it was enough. He first heard Chollet during 2020, on a podcast walking through the 2019 paper *On the Measure of Intelligence*. He parked it. As Zapier shipped more AI product, he started reading evals to see where the technology would tap out. Most benchmarks were racing toward human-level performance. ARC, from 2019, had done the opposite. It had not reached humans, and progress was decelerating.

He sees two popular definitions of AGI. One school says it cannot be defined, so we should not try. The other, written into the OpenAI–Microsoft deal and possibly coined by Vinod Khosla, says AGI is a system that can do the majority of economically useful work humans do. Knoop thinks that is a fine goal, and that current architectures may be within spitting distance of it. He also thinks it says more about what most people do for work than about what AGI is. Chollet’s version is shorter: AGI is the efficiency of acquiring new skills.

The thought experiment is the contrast with games. AI has beaten humans at Go, chess, poker, and Diplomacy for years. You still cannot take a Go system, feed it new data, and have it win at chess. Researchers tear it down and start over: new algorithms, new architectures, new data, often new scale. People are the opposite. Sit someone down, teach a new card game, and they are playing in an hour; a new board game takes a couple of hours. That, Knoop argues, is general intelligence: gaining skill quickly enough to handle a novel task. ARC tries to measure it. Every task is new. A private test set has been seen only by a handful of people who checked that the puzzles are solvable. The point is to tell memorization-heavy narrow AI apart from a system that can acquire skill at test time.

Efficiency, in Chollet’s terms, is how far a system can go from a small set of core-knowledge priors — goal-directedness, objectness, symmetry, rotation, the sort of things that show up early in childhood — into a wide task space. Weak generalization only handles tasks adjacent to the training data. Strong generalization attacks a much larger field of novelty from little data. The grand prize line is 85%. State of the art at recording was 39%. Kaggle enforces no internet and one NVIDIA P100 for 12 hours, so competitors cannot call GPT-4o, Claude, or Gemini. The compute cap exists because without it you could define AGI as “acquires skill, efficiency optional” and brute-force every program. Humans do not do that. They intuition-pick three to five candidate patterns and check them. Prior contests gave two to five GPU hours; this one gives 12, and every current leading method maxes that out. No internet is more practical: less cheating, less contamination, less leakage of the private set, so that when someone does hit 85% the claim is believable.

Frontier labs, Knoop says, talk as if more scale and more data will get us to AGI. He does not think that is what most of them believe internally; they are working on new ideas. It is still in their interest to push a market story of “scale is all you need, don’t compete with us.” Those competitive dynamics have pulled attention, investment, and talent away from exploring new ideas. If new ideas are required — and he thinks ARC Prize shows that at least some new idea is required — the field has been heading the wrong way. Frontier research has gone closed. The GPT-4 paper shared no technical details. The Gemini paper shared none on its long-context work either.

That closure sits in contrast with how we got here. Knoop traces a six- or seven-year open chain from Ilya Sutskever’s sequence-to-sequence paper at Google, through work that moved in and out of the company, to Alec Radford and back to Ilya at OpenAI. That path only existed because of open sharing. One goal of ARC Prize is to push against the closed-research equilibrium. A host notes that the foundation-model companies, at least in public, are walking similar, clearly defined paths. Whatever works for ARC-AGI, Knoop agrees, is going to have a different shape.

His shortcut on language models: they do very high-dimensional memorization, then apply those examples in slightly adjacent contexts. He does not want to throw them under the bus. They are economically useful — Zapier is an existence proof — and something “very magical” happened. Their limit, which ARC makes visible, is composing training data at test time to solve tasks they have never seen. Score history makes the gap concrete. A 2019–2020 Kaggle contest set a 20% baseline. Getting from 20% to 30% meant researchers staring at the public set, hand-writing a domain-specific language of the transformations people do in their heads, then brute-force searching combinations of those programs. Getting from 30% toward 40% is Jack Cole’s method: a code-oriented open-source model, fine-tuned at test time on permutations of the unseen puzzle, until it writes a program that fits.

Knoop suspects the ideas already in the air, ensembled, can get to about 50% without a lot of new invention. The 85% solution looks different. Instead of a hand-coded DSL inferred from the public set, he wants a deep-learning-guided DSL generator that writes the language on the fly, learning from past puzzles and applying that to future ones — the way humans get faster as they work through the set. Pair that with a deep-learning program-synthesis engine that does not enumerate every program, but learns which handful of traces to try, then checks them deterministically. That is the introspection he keeps returning to: people do not brute-force the puzzle. They pick a few likely candidates and test.

Code generation and program synthesis sit under every method Knoop just walked through, and a host flags how general that is. Program synthesis is exacting. That is why Knoop thinks a real ARC solution would be useful almost immediately, unlike a decade of toy puzzle benchmarks that collapsed as soon as scale arrived and taught us little about building products. The question he has been getting since launch is whether ARC is next in that graveyard. His answer is the application layer’s actual bottleneck.

Zapier launched AI bots a couple of months before the taping, and adoption surprised him. He describes AI automation as concentric rings of use cases. Users will let a fully hands-off bot run only where being wrong is not catastrophic: personal productivity, team workflow, places where 90% is fine and a few days of prompt wrangling to reach 95 or 99% is acceptable. Expanding into moderate- and high-risk autonomous work needs something the field does not have — user confidence in what the system can and cannot do exactly.

Classic Zapier is a deterministic engine: once you set it up, it does the same thing every time, which is also what makes it fragile and hard to build. Pure language-model systems flip the trade-off. You steer them in natural language, but accuracy is inexact, so trust stays low. Hallucination, accuracy, and consistency are the number one problem limiting real deployment, because they destroy user trust. An 85% or 100% ARC solution would be a program that generalizes from simple core priors to unseen puzzles with exacting reliability. Knoop wants that program in the builder’s toolkit.

Two weeks after launch, the contest had trended on Twitter twice, sat as the number-one Kaggle competition in the world, and passed a million social views. Historically, the people who work on ARC have been outsiders. Lab42 in Switzerland ran a smaller contest, ARCathon, the year before: 300 teams, nobody beat it, and to Knoop’s knowledge they were individuals with backgrounds in mechanical engineering, game programming, or physics — curious people, not big-lab researchers. He thinks the breakthrough is more likely than not to come from someone who thinks a little differently and can cross-pollinate ideas across fields. That is one reason he put as much money into the prize as he did. Progress was idea-rate-limited, and awareness is how you raise the idea rate.

On Twitter he has watched two camps form, and he wants both. One agrees with the mission and wants to find new ideas. The other wants to prove him wrong: language models plus scale are enough, and off-the-shelf technology will farm the benchmark. To make frontier models usable as instruments, the team was launching a secondary public-task leaderboard, with about $150,000 in a reproducibility fund, scored on the 400 public tasks and verified locally. It is not the official 2024 contest, which keeps the private set and the no-internet rules. Ryan Greenblatt showed up days after launch with an outer loop around GPT-4o: sample programs and reasoning traces, test them against the demonstrations, keep the one that works. At recording that sat in the low 40s. Someone might call it proof that scale is enough. Knoop calls the outer loop itself a new idea, and notes that generating those 8,000 traces cost about $10,000 — a technique that was not practical three or four years ago.

Why aren’t the big labs on this? A handful tried years ago, with weaker models. Knoop would like ARC-AGI on model cards and says he will work with anyone who wants to make that happen. After he came back to Chollet’s work he surveyed researcher friends in the Bay Area. About nine in ten knew who Chollet was. One or two in ten had heard of the ARC eval, and half of those had it confused with one of the five other AI benchmarks also named ARC. When he asked Chollet why the man is famous and the test is not, the answer was that it is hard. Benchmarks get famous when someone makes progress, publishes, gets cited, and a snowball starts. Four years of slow, even decelerating progress makes a lab researcher glance at ARC, call it a toy, and go back to a million other evals.

Prizes, Knoop argues, are outrageously effective only in a narrow band: the problem is small, it is idea-rate-limited, one person or a small team can break through, and the result is quickly inspectable and reproducible. ARC checks every box. That is why he chose a prize rather than another paper.

Past curiosity, a host asks what the world looks like after AGI defined as efficient skill acquisition. Knoop’s near-term picture is not sci-fi. A system that generalizes with exacting accuracy from a small set of priors is a direct attack on the trust problem that rate-limits real-world AI today. The application layer gets much better at consistency and much lower hallucination, and people use it in a less fettered way because of how it is built, not because a chatbot got a little more fluent.

What gets stacked on top of that is unknown. He expects a gradual, incremental rollout with a lot of contact with reality, which gives society time to update on what the systems can and cannot do, and to decide where not to deploy them. That is also why he has been a proponent of open-source progress on AGI through the prize. He does not buy the mythical story of a very bad outcome once we hit superintelligence. It is a theory-driven story, he says, not one grounded in empirical evidence.

Good policy, in his view, looks at what systems can and cannot do and then decides. Anything else cuts off potentially very good futures too early. That is what he thinks a lot of early AI regulation is doing: pausing on the logic that the downside is so large we should stop here, and in the process trimming off every good path. The reason that bothers him is practical. The field still needs new ideas from researchers, students, young people, and labs, or there is a real chance we never get the useful AGI we actually want. Advent, as he sees it, is much less likely to be a moment in time than a stair-step of technology built on past technology, with many chances to update beliefs.

Asked when ARC will cross 85%, Knoop quotes Zapier’s first data scientist: the longer something takes, the more you should update that it will take longer still. Coming into the year, past track record said at least three or four years to the grand prize. Two or three weeks of new work changed the nearer forecast. He thinks 50% during the 2024 contest — it ends in mid-November — is quite likely. Hitting 85% in that window would surprise him, in a good way.

Why now, after five years of failure? If deep learning is part of the answer, the world has three or four years of experience building and scaling those systems, and much more compute, which has pulled some methods into a practical cost range. Greenblatt’s public-leaderboard run is the exhibit: $10,000 of GPT-4o traces would not have been a serious experiment a few years ago. Then Knoop flips the question. The reason they launched the prize is the risk that it is *not* now — that LLM-only attention, closed research, and application-layer heat have steered people away from the new-idea set. The “why now” he wants is simply that ARC finally has enough attention.

Will LLMs be part of the 85% solution? He has seen research suggesting transformers, as an architecture, can represent very deep deductive reasoning chains at 100% accuracy. The missing piece is the learning algorithm: backpropagation does not teach a network to do that, and nobody has found the right outer loop around the transformer to run program synthesis or DSL generation. He is confident deep learning will show up in the grand-prize winner, and that a purely deterministic program will not. He is less bullish that LLMs in their current form are the whole system. They are more likely a subcomponent. The search space of alternative architectures is rich — LSTMs, CNNs, RNNs, state-space models, and others — each with different properties, even as new ASICs chase transformer throughput.

To claim the money, a winner has to put reproducible code in the public domain, on both the public board and the official contest. Knoop wants open progress that other people can stair-step on, not a private million-dollar acquihire. In the lightning round he names Chollet and Rich Sutton as the two people who have shaped his thinking; both published in 2019, Sutton’s *The Bitter Lesson* with one asterisk: scale has been applied to search and learning at inference and training time, but architecture search is still a human-handcrafted story. His most contrarian view is the thesis of the hour: scale is not all you need; new ideas are needed. He pays for ChatGPT, Perplexity, and Claude, and says the last six months of using them as tutors on unstructured text taught him more — including starting to train models himself — than the early Zapier years. The thing he wants pulled forward is a future in which AI is not forever rate-limited by the human in the loop, and can invent, discover new drugs and physics, and even figure out how to build AI, alongside people.