An Agent Posted Five Tweets — and 5,000 People Didn't Notice

In January 2024, Cognitive Revolution host Nathan Labenz did something a little reckless: he handed his Twitter account to MultiOn, a web agent from Div Garg's company. The prompt was specific. Search the last ten tweets and remember them, then hunt AI news on up to three sources, skip anything dull or already posted, and write no more than five small, informative tweets. It worked. Labenz rates the posts below his usual standard, but none of the 5,000 people they reached questioned the origin. Garg offered immediate beta access to anyone willing to try the same prompt; his handle is divgarg9.

This is Garg's second appearance on the show. He first came on in July 2023, when post-GPT-4 agent fever was at its peak. Since then the field has slumped into a trough of disillusionment, and many agent startups have gone heads-down. MultiOn kept iterating in public, posting videos and slowly widening a private beta that Labenz still describes as limited relative to a human assistant, even as the successes get more impressive.

Asked for the current state of agents, Garg is blunt. Everyone got hyped about GPT-4; in his view it is not that powerful. It can chat well, and maybe write some code, and that was enough for people to talk about AGI. What actually arrived was conversation good enough to pass a vague Turing test, plus a talent for hiding logical mistakes. The coding version of the same failure is familiar: the model writes something that looks great, then you spend the rest of the day on bugs. His analogy is a paper that looks fancy, packed with math, until you dig in and find that none of the theorems hold. The deep logical connections are still not there.

Labenz paints a rosier picture of coding. He calls his method "coding by analogy": give the model an example from docs or an existing snippet and a target, and let it move from one to the other. A couple of iterations is normal, but it still feels like a major unlock, not a wash. The two of them are drawing the same boundary from opposite sides. Pattern matching is strong; strict reasoning is weak. Meanwhile, walking a browser to checkout has been a year-long humiliation for most products.

Why is clicking through a site so much harder than chatting? Garg says it is a combination of everything, starting with training. Current models were not trained on the process of operating in a world environment — code, APIs, or the web — so they cannot even represent the state they are in. They take actions that look meaningful without understanding the environment's edge cases. They are not grounded.

Humans have the same problem on a new site. Garg has been inside AWS more than a thousand times and still cannot find the setting hiding in a dropdown. People learn by going, failing, succeeding, and absorbing the experience fast — reinforcement learning on the fly. That loop is what agents lack. If they could adapt to a new website automatically, ground themselves, and discover techniques, he thinks success would move from 90% or 95% toward something close to 100%.

The next few months of MultiOn research point at online learning: train an agent directly on the live internet. They do not want to "take down the internet," so they would start with reversible work such as research, or stop the agent before it places an order. Garg is collaborating with the first author of DPO at Stanford and with other academics, trying to take twenty years of reinforcement learning and imitation learning that never left the lab and actually run it in industry. RL, he thinks, will not help chat much. It will help agents, because agents live in explore-exploit loops over long processes.

Labenz had expected agents to "start to work" by the end of 2023. They have not, and one of his pet theories is that vision, especially GPT-4V, rolled out slower than the March 2023 reveal implied. Garg's 2024 plan is dual: pick off low-hanging tasks that can drive adoption now, and keep grinding on the hard research. The company goal is agents that add value to everyday life, even if that starts as one thing done extremely well. He wants a new breed of researcher thinking at the trajectory and process level, not bottled into supervised next-token prediction.

OpenAI has told this story more than once: years ago they tried a web agent with reinforcement learning and got nowhere, because successes were so rare that there was no reward signal. Labenz asks whether Garg is now moving away from a pure language-model stack toward extra models that specialize in actions. Garg's correction is that there is nothing inherently linguistic about Transformers. He did research on action transformers at Stanford and wants to go fully multimodal, optimizing whole trajectories instead of next-token, next-state steps — new loss functions, maybe backprop on the whole process.

The missing ingredients last time were language models themselves. They encode too much general knowledge to start from scratch, so MultiOn begins with open-source models as encoders of human intuition and adds capabilities on top. The other missing piece is that nobody really understands how to scale RL even now. The working recipe is a model that already gets you to about 90% without RL, then RL on top. From 0%, it is too unstable. DPO is the first of a category of algorithms that might finally unlock that stack for agents.

Users bounce between two failure modes. Ask MultiOn to Google something, and you have only added a step. Ask for a branching novel of logic, and it breaks. Garg's calibration is narrower. The product is currently very good on a single website: go to Amazon and put five books in a cart, order on DoorDash, get Instacart ingredients for spaghetti, send an NDA to an email, book a 2 p.m. meeting and invite someone. Short, to-do-shaped tasks on one site, short context, one-off — that is the high-success band.

The next problem is composition. Check the calendar for the next event and call an Uber if it is in person; find target profiles on LinkedIn and cold-email them from Gmail. The hard part is moving context from one site to another. After that comes scheduled work: order coffee every morning automatically. Garg's mental ladder for the year is mini-tasks first, then longer ones, then combinations, then running them in parallel. Users should start at the bottom of that ladder on purpose.

Labenz noticed a recent MultiOn update that seemed to handle much more context and asked how they think about windows. Garg's observation runs the other way. GPT-4 and Claude both get confused if you dump too many random things into context; they lose focus and start making mistakes. MultiOn has seen the same: more notes and user trivia, worse action selection. Minimizing context somehow restores logic. Context management, he says, is currently the biggest lever they have.

They mix retrieval with an external memory and borrowed an idea from MemGPT: let the model fetch. One of MultiOn's actions is store-in-memory or retrieve-from-memory, so the agent can decide, like a CPU, that instead of clicking or typing it should stash the current content or pull something it needs before continuing. The model is managing its own memory. Average prompt size at each inference step is not more than 5,000 tokens.

Sam Altman had just told a Y Combinator batch to plan for GPT-5 and early AGI, which raises the obvious question: does all this scaffolding become worthless when a better model arrives? Garg's bet is that efficiency always matters. If you already have the most compressed, information-dense representation of the environment, a 10x better model still helps you more than it helps people who never did that work. They try to project GPT-5 along context, reasoning, and speed, then design so those gains compound rather than erase the stack.

What users actually run are MultiOn's own fine-tunes on open-source architectures, not a foundation model from scratch. GPT-4 is still in the loop for planning. The rest of the system is plug-and-play: drop in Claude or a newly fine-tuned model, run the same battery, read the score. Prompts started on OpenAI's format and were carried into the trained models, which Garg says has given them decent backward compatibility. They have squeezed about as much as they can from GPT-4; next is a better model or more fine-tuning.

The internet is dynamic, so a static benchmark rots. A lot of MultiOn's eval is real-world and manual: can it actually call an Uber, can it actually get a burger delivered. Numbers that do not survive that test are decoration. They also run automated scenarios — gather information against a known source, find shoes in a given size, match a complicated DoorDash order — and let another model judge whether the final state is correct. Labenz is glad they have not "transcended" watching raw logs. The behaviors are still too weird to fully standardize.

Most agent teams would max success rate and ignore cost and latency. MultiOn wants a product, not a paper, so snappiness is a first-class goal. The value proposition of an agent is that it does things for you; if it is only as good as you and no faster, why use it? They measure against human speed and want at least 10x. Compression creates a positive cycle: the same tricks that make the agent fast also make it more accurate. In their comparisons with other agents, working faster and working better have gone together.

Vision is the obvious example. GPT-4V's low-res image is billed as 85 tokens, an order of magnitude cheaper than stuffing bloated, autogenerated HTML into the model, and closer to how the page is meant to be read. It does not close the loop. Seeing that you should add to cart does not tell you the pixel coordinates of the button, and current vision models are not trained to emit those coordinates. You still need segmentation or captioning to propose elements the policy can click. MultiOn mixes language metadata from the HTML with images rather than betting on screenshots alone.

Each chat-box line — "I'm going to your profile," "I'm scrolling for more tweets" — is an inference step. Simple tasks take fewer than 20 steps; an information-gathering job like the tweet-research prompt can run about 100. Cost per step is not more than 10 cents on the high side, so ten steps is roughly a dollar. With more efficient hosted models they can get to two or three cents a step, which puts a hundred subtasks in the $2 to $3 range — against a human doing the same work at $15 an hour or more. At scale they cache aggressively.

A Voyager-style skills library is due in about a month: learned procedures the agent can reuse instead of recomputing every click. The web changes, but not that much from day to day. Garg says a lot of production systems are quietly migrating off GPT-4 because token prices cannot support a consumer product with millions of users, and vision is even worse. Cursor and Khan Academy get cost and latency wins from OpenAI dedicated instances plus prompt caching, which starts around a six-figure annual commitment. MultiOn may use that, but they are also building their own model-agnostic cache at the edge so they are not stuck with one provider.

Why does GPT-4 still lead MMLU by seven or eight points? Garg's cake metaphor: OpenAI has the world's best chefs, people who have been training models for five to ten years, plus private high-quality data and a full human labeling pipeline they do not broadcast. Models are a distillation of human knowledge. Whoever collects that knowledge at scale, filters the noise, and trains on it wins; open-source data alone no longer builds the best model. At an agents hackathon, Andrej Karpathy said OpenAI had spent years obsessed with the language model itself and had not really explored the scaffolding the community was inventing. That, Garg implies, is the gap MultiOn is occupying.

The next unlocks, in his view, are architectural. Attention is quadratic, which is why nobody actually uses GPT-4 Turbo's 128k window at full fidelity — you end up with approximations. Linear or sub-quadratic models such as Mamba would make long sequences usable, with a side benefit for biology if you could attend over DNA. He also likes diffusion-style planning: a high-level sketch of the next ten steps, refined down a tree into execution, closer to how people write programs than left-to-right tokens. Later in the year he wants parallel sub-agents behind one chat box, scheduled like processes on an OS, but only after a single agent is a reliable executor. Otherwise you spend the day re-delegating failures.

Nirvana, as Garg sketches it, starts with one-off short tasks, then long tasks, then combinations, then parallel work. The interface should leave the laptop. MultiOn already has a partner API in private beta for server-side orchestration, which is how you would order a burger from a phone. He wants a Siri-like loop: talk to an AI and it just happens — "what Siri could never be." The Chrome extension was always the cheat code, because it piggybacks on the user's logged-in sessions and skips the hardest part of browser automation. Mobile has no such gift.

He will not spill the mechanism, but he already uses MultiOn from his phone on his LinkedIn account. It does not store his password and still authenticates well enough to send connection requests. That path, he says, has been validated over months and will ship soon. Labenz's picture of 2024 is walking around, delegating on the fly by voice: connect with Div, order dinner, stop carrying a mental to-do list back to the desk. Garg sees executive assistants as natural early power users because they already know the chores and do not need to be sold. For 2024 he calls the product a complement — doing what humans do not want to do or are not good at. Replacement of professional help, if it comes, is 2025 or later, maybe 2025-plus. The near-term win is zero-to-one for people who cannot afford an assistant.

That instinct has a personal root. Garg grew up in India, where cheap household help is commonplace, so the idea that someone — or something — should do your chores feels natural. Labenz compares it to Vivek Natarajan's medical-model work at Google, aimed at places without doctors. On jobs, Garg does not dodge substitution. A lot of roles exist only because the technology is not there. Typewriter operators had a job nobody wanted; computers ended it and created more work of a different kind. "Just because the technology is not there, we are using humans as a substitute." Digital chores will go the same way. New jobs will show up in managing, teaching, and programming agents, the way computer science was not on anyone's list when the machines arrived.

The product already has a "teach me" mode: demonstrate a task in the browser and it becomes training data, with almost no onboarding because everyone already knows Chrome. They will not train on paying users' authenticated accounts. Gmail autocomplete leaking one person's mail into another's is the cautionary tale, and the New York Times lawsuit is a sign the world is getting less naive about training data. Public data plus volunteer testers is the mix. On money, they are already charging some API partners and plan a free consumer tier plus Pro. Labenz's company Waymark currently scrapes small-business homepages with a dumb crawler, then dumps text into GPT-3.5 — and drowns in Facebook icons. Garg's pitch is that the API should be a no-code layer over Playwright: an English instruction in, JSON or on-site actions out.

Are websites adapting? Still early, Garg says. Inside the Bay Area people know; outside, agents are barely on quarterly roadmaps. Businesses ping him every day asking about marketing, outreach, onboarding, and CRM flows with terrible UIs, so the mood is still the bright side. He expects something to go badly wrong — maybe an AI virus — which is why MultiOn wants to be the agent a site is willing to let in. Labenz has been red-teaming a calling agent by having it phone him as a kidnapper demanding a million dollars. It complied with zero guardrails. He has given the developer a deadline before he names them in public. The industry, he argues, has to self-regulate or it will be regulated.

MultiOn has already built prompt-injection detectors and classifiers, even though they have not been hit in the wild. After the model proposes browser actions and before those actions run, a verification step checks whether they are harmful or wrong — another model call in the loop. They can change agent behavior per site, cut off a malicious pattern immediately, and patch the way OpenAI patches ChatGPT jailbreaks within a day. Garg's minimum kit for everyone else is almost boring: put OpenAI's cheap public moderation API in the chain, and add a couple of prompt lines that forbid harm. Labenz is less impressed. His GPT-4 spear-phishing red-team prompt did not trip the moderation endpoint, and Sander Schulhoff's HackAPrompt contest, with thousands of players, found that a clever adversary can usually walk around a template that says "don't do bad things."

The last gap versus a human, if a human is AGI v1, is planning. Language models are sequence predictors. Ask one to play chess and win, and it will not, because it cannot hold a plan. Garg expects this year to bring Monte Carlo tree search and AlphaGo-style planners bolted onto LLMs. GPT-4 can already fool average people in chat and even in voice; it cannot fool an expert, who hears the hallucination immediately. It has learned the shallow front of the human brain, not the deep work. Last year's OpenAI board fight, he thinks, was a preview: every technology revolution polarizes people into accelerationists and people who want to bomb GPU clusters, and those swings will get wilder as systems approach human capability.