A 70B Model Booked Dinner 20% of the Time. MultiOn Took It to 95% in a Day

Div Garg opens with a market that no longer has a sole winner. OpenAI, he says, has lost much of the lead it held with GPT-4, when nobody could catch up; the frontier now looks homogeneous, with everyone close. Disruptive technology takes a long time to catch on, and he thinks online interaction is only at the start of being disrupted by agents. Humans are already good at navigating websites through UI. In theory, he argues, AI can become as good, or better.

Nathan Labenz brings Garg back for a third appearance on *The Cognitive Revolution*. The first conversation, in mid-2023, landed while BabyAGI-style projects were handing GPT-4 a pile of tools and almost no guidance, then stepping back to watch. The amazing moments were outliers. On average, the step-by-step error rate — including on mundane microtasks — was too high to string long sequences together, and models that can self-critique still get stuck on obstacles a person walks around. Much of the next 18 months went into scaffolding: intelligent workflows a human designs, with the model filling the bits that need judgment and no freedom to choose its own adventure.

As of January, MultiOn was still among the most bullish on open-ended agents. The company has added scaffolding and domain-specific fine-tuning, Labenz notes, but the product still takes arbitrary natural language and tries to fulfill it. Progress in his testing was obvious, and in some contexts MultiOn claims human-level performance. The system as a whole is still not a substitute for a human assistant. That gap — a 70B instruct model under 20% on OpenTable, GPT-4o a little over 60%, AgentQ around 95% after a day of stacked techniques — is the result this episode turns on.

Asked to tell the story of agents since GPT-4, Garg refuses the eulogy. He still thinks the field is early, closer to the first wave of the internet than to a failed boom: glitchy, slow, unused by most people, with quality and reliability still ahead. The GPT-4 moment was a promise of automatic capabilities. The early versions are showing up in specific verticals and some business use cases. Most people, he says, have not yet built an agentic product, and exploring that fully will take more time.

His own agent lifestyle is not pervasive, and he says so. Shopping is the biggest case — Instacart and groceries — plus calendar invites and bots on LinkedIn and Twitter. A lot of his scheduling now runs on agents, as does some online shopping. Nothing in that list, he admits, is everywhere. The industry has a handful of working habits and almost no default agentic applications.

Labenz puts the fork on the table. The first wave — BabyAGI and friends — gave GPT-4 an open-ended environment and hoped. The compensating move was to put agents on rails: prescribed workflows, intelligence only at the steps that need it. MultiOn's preview still feels open-ended. Garg agrees that on-rails is the right way to start if you already know the use case, especially in enterprise and B2B SaaS. Linear, constrained workflows are the right paradigm there. Over time you can open the aperture.

MultiOn is trying to live in the middle. The company has surveyed more than 3,000 people, and Garg says consumer behavior changes with demographics, age, and where you live. Fully constrained workflows would mean building millions of them, which is not possible. Fully open-ended is no better: Anthropic's computer-use API is very general-purpose, and it is very hard to find a stable utility for it. Building a fully general model, he adds, does not look like something that pays off for a startup in the next six months.

Behind the open-ended chat, the architecture they have landed on is user choices. Garg's example is flight booking. You do not want an agent to book a random flight and waste thousands of dollars without asking whether evening, nonstop, or a particular airline is acceptable. Preferences matter. A user might say, book me a flight to Paris this weekend; the agent fills in the blanks and surfaces the implicit choices. The product problem is not only actions. It is how to capture preferences and keep a human in the loop.

An earlier MultiOn build let users teach the agent — demonstrate or correct a skill when it went wrong. Garg says that method collected millions of trajectories and paid off some. The trouble with crowdsourcing is quality. Tesla is his example: the company crowdsources driving data constantly, then has to filter good from bad, which means a filtering pipeline and a lot of AI logic around it. The data is noisy. The other path is high-quality annotation and expert traces, a paradigm he thinks the frontier is pushing toward for general-purpose agents.

Labenz's counter is cost. High-end annotators are expensive and scarce. Why, he asks, is nobody paying him to watch him use a computer? He wants a general observer, maybe with a mode where you talk through what you are doing, because chain of thought is rarely said out loud. Clean it, anonymize it, earn trust first — and at current annotation prices he feels he should be getting at least $1,000 a month. Garg says that market exists now and did not, even a month or two ago. Generic computer-use data is not that expensive: enough people will generate it cheaply, especially if you outsource collection. The price jumps if a company wants *Nathan* — what it looks like when he uses his computer.

Labenz reaches for MindEye. The first version needed something like 20 hours of fMRI per person to train a patient-specific model; MindEye 2 mapped many brains into a shared space, then needed about an hour of one person's data to fine-tune. He expects computer-use data to follow the same split: bulk generic traces at a low price, individual data on the margin. Garg agrees. Common patterns can be outsourced. Private, personal traces are a market that barely exists and that people will pay a lot for if you are willing to live with the loss of privacy. In practice MultiOn works with labeling companies, has tried MTurk-style discrete browser tasks — he suspects Anthropic is doing something similar — and hunts for data in the regimes where the agent is most deficient.

Self-play is the more radical path, and Garg says the missing piece is a high-quality environment reward. Chess and AlphaZero have one. Coding can get a numeric score from unit tests. Real-world computer use does not. You end up training a second model to give a proxy reward, and any metric can be cheated — null strings that score infinity, the DeepMind 2017 clip Labenz still sees of a little boat circling the same patch for infinite points. Human feedback can supply a proxy. It is a harder problem than putting self-play on a game.

Production at MultiOn still leans on Llama 70B as a compromise between speed and reasoning. Garg is moving toward inference-time compute: chain of thought at inference, more like the o1 bet than the 2023 habit of throwing parameters and data at training. At current scale he thinks the base model barely matters. Recipes and data look similar, leaderboards are tied, and OpenAI's GPT-4-era gap has closed. A 10x jump in parameters might let a different architecture shine. Until then, the trick is the application. o1, at least in preview, he calls very good at math and not very good outside it. Throw less compute and it may be worse than GPT-4; throw more and it may be much better. The product question becomes how much latency and resource you will give the model.

AgentQ is the vertical-specialization result. Two environments: WebShop, an academic Amazon-like shopping sandbox, and live OpenTable. Llama 3 70B Instruct comes in under 20% on OpenTable; GPT-4o is over 60%, still not a product. Generic models, Garg says, are trained on generic intent and are not super at one thing. Llama has probably never been trained on these tasks; GPT-4 may have more browsing data because OpenAI still has a browsing tool. Neither is fine-tuned to the vertical, so accuracy leaks away. MultiOn collected data, layered DPO and reinforcement learning from human feedback, and added Monte Carlo tree search so the agent can explore routes on a site, keep what works, and discard what does not. It took a day. It is a self-improvement cycle. The jump was about 4.5x, to around 95%, and even they were surprised.

RFT — reinforcement fine-tuning, an older algorithm they used as a baseline — moved the needle only so far. DPO is the intuitive next layer. Supervised fine-tuning shows the model optimal data and gradient-descents toward it. DPO also uses negatives: gradient descent toward positive behavior, gradient ascent away from the bad, so the model's mass cannot smear all over the map while still covering the target circle. MultiOn's modification in the AgentQ paper is trajectory-level DPO. An agent takes many steps before an end state; scores attach to whole trajectories, closer to PPO, not to a single token. Original DPO, Garg says, was simpler: a plus or minus on a generation, applied across tokens, maximizing the probability of good generations and minimizing the bad.

The biggest bar on the chart is Monte Carlo tree search. AgentQ, as Garg defines it, borrows Richard Sutton's bitter lesson: the thing that seems to work is search plus learning. He points at Noam Brown's Diplomacy work at Meta and the same combination now at OpenAI. Search for agents, he thinks, has been underexplored. MCTS lets the agent wander the environment with extra entropy at the start, so it does not collapse onto one narrow path it already knows. Successful trajectories become positives; failures become negatives; those feed the learning loop. Repeat until the model has seen the environment split and knows how to reach the goal. As long as the feedback signal is high quality, he says, you can keep making the model better.

Labenz will still bet on himself against the MultiOn agent. Benchmarks keep showing AIs beating humans, and if you only looked at scores you might think AGI was already walking around. Garg's theory is mock versus real. A benchmark is a narrow domain: WebShop has maybe five or six environments, and MultiOn trains site by site until the model is very good at that site. A general-purpose agent on the whole internet is a different scale of problem — more data, more feedback, more compute. On a vertical you care about, beating average human accuracy is a real result. Generalizing from there is the unsolved scaling question.

He is not gloomy about transfer. Amazon, Target, Best Buy, and Walmart share a search bar, a catalog, a product page, and checkout; train on one and similar domains take a lot of positive transfer. The internet was made for humans and is not that diverse: dropdowns, drawers, navigation bars, text fields. Train on enough sites and he expects a scaling law that generalizes to sites the model has never seen. Fine-tuning practice follows the data you have. LoRA is efficient and, in the 100,000-token range, the better use of limited data because there are fewer parameters to optimize. Billions of tokens justify full-weight training, which he frames with Chinchilla scaling laws. MultiOn optimizes for speed: a couple of days of training on AgentQ, ship it. Purist research can spend six months and hundreds of billions of tokens on a general agent. Productizing a narrow agent is much easier unless you want to be a foundation-model company.

Vision is a hybrid pipeline. DOM meta tags and ARIA labels are already optimized for bots; visual data carries layout and state; MultiOn wants both. The agent's screen is not the human's. It can click invisible elements. A ten-page site it may "see" without scrolling, which throws off anyone who expects human behavior, so they have to constrain it. Humans also catch elements that are obvious on screen and opaque to the model — complex JavaScript, iframes. Garg says the major products have gotten more robust over the last couple of months. Labenz's TestFlight still found the gap. The day before Thanksgiving in Detroit, he asked the in-development app for Lions tickets and Michigan–Ohio State tickets. The agent understood the query, searched, sometimes went through Google, sometimes went straight to a ticket site from prior knowledge — and then, with tickets on the screen, said it was having a hard time finding them. An Instacart Thanksgiving shop stuck on account creation until Labenz pasted the email code into the chat.

Ticket sites have been fighting bots for years, which may be part of the stall. Labenz's larger question is who will invite agents in. Instacart might want to be an early adopter of whatever auth paradigm arrives; Ticketmaster or StubHub might want to ban agents because they eat margin. Garg says authentication is getting more mature — Anon is building agent identity for browsing — but it is not yet a plug-in-and-go market. Long term he expects merchants to cooperate, because agents can help revenue and users. Short term is harder: spammy behavior, trust, backlash. Disruptive technology takes time. He thinks we are at the start of the era in which online interaction gets disrupted by agents, with an initial wave of "this is dangerous" before the win-win.

Looking six months out, Garg is betting on verticals: restaurant reservations, travel, scheduling. AgentQ was a first step toward agents you can train on a niche — not a brittle Playwright script that breaks when the UI moves, and not a fully general model. Reliability, human-in-the-loop, and the rest of the product are "ten or more other things" besides the agent. The same stack could unlock business process automation. UiPath-style RPA, he says, is brittle, slow to onboard, and dependent on specialist engineers. If ordinary people can build resilient automations, that is a large unlock. Everyday use is still the company's north star.

Should a Kayak or OpenTable partner skip the UI and call an internal API? Garg reaches for self-driving. Building a special lane with sensors for autonomous cars is simple engineering and a ten-year infrastructure-and-politics project. Startups chose to drive on ordinary roads. A special internet protocol for agents may arrive, and MultiOn has looked at it; until everyone agrees, the bet with AI is that whatever a human can do on a UI, a model can learn to do as well or better. Sites already have backend APIs. Exposing them as an API-first business, he argues, collides with product teams, frontend teams, identity, and SOC 2. Engineering-wise it is possible. As a business and security pattern it complicates the landscape a lot.

On cost, speed, and reliability, Garg splits the world into soft and hard boundaries. Chat is a soft boundary: nothing irreversible. A purchase or a trip is a hard one. Users will wait — even 20 minutes — and pay for more reasoning if the irreversible job is done correctly every time. Reversible work can run on a small, fast model. Irreversible work needs verification and fewer edge cases before the agent actuates. Detecting which is which is something MultiOn has spent time on. Labenz, disclosing that he is a very small investor in the company, then asks the bitter-lesson question: Claude already has computer use, OpenAI is rumored to be next, and an agent company that has not raised $10 billion is in the blast radius.

Garg's answer is niche. Perplexity picked search and treated hallucination as the one problem to fix. Cursor picked the coding IDE. Computer use is a capability, not a product. Frontier labs with hundreds of billions of dollars do not want a narrow domain; they want to throw compute at something general. Someone still has to turn that block into a product experience. 2024's technology was too early: MultiOn was among the first to offer any agentic capability, and you cannot build a cart before you invent the wheel. He expects early 2025 to bring an explosion of applications that has not happened yet. Agent apps are still the rarity. He will not say whether MultiOn will sit on Claude computer use; the company is close to Anthropic. The method he likes is product-focused research: start from a missing block in a real product, then do the R&D to fill it, instead of researching in a vacuum and hunting for a use.

MultiOn is hiring researchers, engineers, and product people. What Garg wants to spend time on next is not another benchmark. It is the interaction: an agent that runs on its own, that you can take over, that you can operate manually, that you can watch and correct so it learns. He calls that a unique paradigm and a hard product problem. Asked to paint a year ahead, he reaches for Jarvis: look up my files, book this flight, handle the boring dentist appointment. People were bullish on that last year, he says, and the technology was not there. Fast-forward to 2025 and those assistants might start working well enough to become mainstream, with an explosion of vertical applications that has not happened yet.

The episode's usable residue is less cinematic. Pick a narrow track before you talk about a general agent, and split the track by whether a mistake can be undone. Do not jump to full-weight fine-tuning at 100,000 tokens; LoRA is the better tool until you actually have billions of tokens and can afford a long train. And do not read a 95% leaderboard as a license to fire the assistant. Ask how far the benchmark sits from the target environment, whether the model was optimized site by site, and whether the failure mode is reversible. Generic computer-use data is about to be cheap. If you hold a vertical's private traces, Garg has just told you they are the expensive side of a market that barely existed two months ago.