Don't Make the Tools Agentic: Flo Crivello on What Production Agents Actually Need

Nathan Labenz wanted a no-BS read on agents before a talk at Imagine AI Live, so he called Flo Crivello. This is Crivello's sixth appearance on The Cognitive Revolution and only his second deep dive on Lindy, the agent platform he runs. Labenz's own framing sets the tension: the world went from BabyAGI and Auto-GPT demos around GPT-4's 2023 launch to a present in which most production "agents" are intelligent workflows, doing fluid-intelligence tasks inside a control flow humans designed. Open-ended, choose-your-own-adventure agents are starting to work — OpenAI's Operator, Claude Code, Gemini in Sheets — but only starting.

Asked for a definition, Crivello said the definition barely matters: you know it when you see it. If you insist, he likes Harrison Chase's. An agent is software in which at least part of the control flow is defined by an LLM. That makes agency a spectrum. The more of the flow the model owns, the more agentic the system, the same way some employees operate inside tight rails and a CEO has almost none. It sits closer to Dharmesh Shah's "anything that helps you get stuff done" than to Amjad Masad's "it's an agent when it decides when to halt." Labenz's worry is that people who are not in the weeds still picture ChaosGPT: a one-line instruction, full autonomy, come back when it's done. That picture, he argued, leads them astray about what is realistic and about how much work it takes just to assemble context.

Crivello would not say Lindies run his life, but he does run a few very open-ended ones. The far end of his spectrum is when you let the model take a lot of the decisions. The very first version of the product lived there, and it overestimated what the models could do. Since then Lindy has backtracked: you can now force a deterministic set of steps. A support agent that should always check a Notion knowledge base before touching an Intercom or Zendesk ticket should not be begged, as he put it, "LLM god," to go look. That check wants to be hard-baked into the cognitive structure. How tight the guardrails are is now a product choice.

His meeting scheduler is still a pure agent: a big prompt of rules and preferences, and within those boundaries it does pretty much anything it wants. Another Lindy wakes every Monday, hunts for new episodes from Dwarkesh Patel, Lenny Rachitsky, and The Cognitive Revolution, summarizes them, and pings him. That one has to be open-ended because there is no single source of truth — YouTube, iTunes, figure it out. Labenz had assumed Lindy's "search the web" primitive might itself be a mini-agent. Crivello shut that down. It is a Google search. There is an agent, and there are the tools the agent uses.

In the exploratory early days they tried agentic tools — sub-agents behind every action — and learned it was a pretty bad idea. Empirically it did not work well. It also made the system hard to reason about: one agent is already hard, two that have to interface is worse. Draw a sharp line, he said. Tools should not be agentic.

That stance does not make him bearish on Google's A2A, MCP, or "smart MCP" patterns such as a coding agent calling a sequential-thinking server. He has been excited about multi-agent systems for a long time. They are just a lot younger. Models were not ready, tooling was not ready, and something like logistics' old EDI protocol — the formal handshake of purchase orders and shipping exceptions that the physical world still runs on — will probably be needed for intelligent communication. It is early, close enough to think about, not close enough to bet the product on.

He already uses a small multi-agent setup in production. Lindy sits in every meeting and takes notes. After a candidate interview that is clearly a pass, he waits until the person leaves and tells the notetaker to pass. That Lindy messages his "chief of staff" Lindy, which knows what "pass" means: do not fire the email immediately, wait a couple of days, and if a recruiter made the introduction, notify them too. It is a toy-sounding example that he actually runs weekly. The delegation itself is one extra step. The orchestration of any single agent, multi or not, can already get gnarly.

Labenz brought up METR's graph: historical task length at 50% success, a seven-month doubling time, maybe four months in a new regime. Crivello is "very AGI-pilled" and still called it dangerous to drive a line across a log graph with so few points. Moore's law and scaling laws have decades and five to ten orders of magnitude. Agents do not. He does not dispute the past two years — GPT-3.5 was profoundly dumb, GPT-4 changed the company, Claude 3.7 and Gemini 2.5 Pro are now fast, smart, cheap, and long-context — and he has high expectations for the next two. He will not sign the exponential.

Where should a new Lindy user aim if they want value they can trust? Crivello reframes task length as ambiguity. If you can write the work as a sequence of steps, it is a low-ambiguity workflow — which is also a pile of short tasks. A surprising amount of real work is two-minute steps in succession, not a 30-second atomic job. Anything you would hand an intern with a Google Doc of SOPs is in bounds. The length of the doc barely matters. What matters is the hardest single step, and even that matters less than people think, because the builder has human-in-the-loop. Toggle confirmation on a risky step and you insert yourself.

Those confirmation clicks are not just brakes. Lindy treats them as in-context reinforcement from human feedback: she learns from the corrections, and she learns quickly. Labenz lit up at that, because getting people to staple themselves to a chair and write gold-standard examples is, in his experience, nearly impossible. Clear instructions most people can bang out. The few-shot set usually lives in someone's head. Compiling it by reviewing live runs is, Crivello said, still the number-one performance driver — much better than instructions.

Fine-tuning is the obvious next question. He is cool on it. Models got so good that the team that used to fine-tune for agentic behavior no longer needs to; open-source also failed to deliver the way they hoped. OpenAI is still pushing reinforcement fine-tuning into production, which he reads as a big company with many things on the stove, not a verdict. The cases he would still fine-tune are narrow, high-volume, and worth amortizing a high fixed engineering cost — reranking inside RAG, a healthcare transcript to a billing code, maybe a slice of what Cursor and Windsurf do. A company-wide fine-tune that "does everything" is, both agreed, the wrong shape.

Weighted by task count, the work going through Lindy is ironic: high-volume, small jobs, mostly email and Slack. Triage, draft, look at an inbound proposal and reject it before it hits a human. That is where Labenz tells people to start — simple, relatively low risk, points on the board. Weighted by credits, the picture changes. Prospecting APIs for lead gen are expensive; Crivello walked through a recruiter Lindy that finds 30 San Francisco engineers at 40 cents a lead, then another few dollars to email them. The other heavy user is what he called a portmanteau of "deep research": consume a large pile of tokens and do something with them. Agents are excellent at that. Humans reading the same tokens are slow and expensive.

His favorite instance sits at the interface between Lindy the company and the outside world. She is on every customer call, every prospect call, every support ticket. At the end of the day she writes a report — pipeline, what customers said, what is breaking in the inbox — and drops it in the general Slack channel. Hundreds of thousands of tokens a run, every 24 hours, the heartbeat of the company. The on-site chatbot Labenz praised is a different Lindy, but her transcripts feed that summarizer. When many people ask the same thing, it shows up in the digest.

Why do most vendor chatbots still suck, then, if this one felt like the future? Inertia, Labenz suggested. Crivello's answer was less mystical. That chat *is* a Lindy. They spent a long time on the platform and then a long time on this particular agent: a lot of prompt, scaffolding that makes sense, the right context from the right tools at the right time, and a good model. He suspects many companies, trying to save money, put a weak model behind the widget. Skip o1 and o3 if the bill scares you; give customers Gemini 2.5 Pro. It is not that expensive, and it performs. The capability, both agreed, is undertapped. Ninety-nine percent of businesses are not yet doing what is already possible.

Assembling context is the other constant complaint Labenz hears. Crivello's lesson is reps until you grow an intuition for the balance between vector search — BM25 and friends — and handcrafted retrieval. Concretely, Lindy's customer-facing chat branches: a billing question hits a billing slice of the knowledge base with a query written for that slice. He also hates being the guy who says RAG is dead, so he said it is limping. If you are not terrified of cost, five or ten billing pages are a couple thousand tokens: dump them in the window. They do not do that for the whole support corpus yet, because it still gets expensive. The practical loop is 80/20: hand-curate the top categories, leave a catch-all search bucket, and shrink that bucket as you watch the agent fail.

People resist that loop even though onboarding a human teammate takes weeks. Agents, he argued, are easier to ramp than humans; the interaction is just less natural, because you have to know the editor. Monitor, catch the dumb answer, edit the prompt, add a step, tweak the knowledge base, rinse. Memory research — HippoRAG, Titans, ChatGPT's mix of explicit saved facts and something vaguer — he reads with interest and then applies the bitter lesson. As models use more context more fully, those systems go moot. You throw it in the window. He wants simplicity because of an engineering rule he likes: you need to be twice as intelligent to debug a system as to design it. If you design at full intelligence, you cannot debug. Academic memory graphs, in his view, were not built under that constraint.

What ChatGPT actually does, as he understands it, is the simplest system at the greatest scale, which he does not think is a coincidence: decide whether a conversation has a save-worthy memory, distill it to a short sentence, inject the pile into the window, maybe score importance, maybe decay with time. Labenz's o3 guess included vector search. Crivello would bet a lot of money there is some, then immediately argued that a naive vector store could not answer "what do you know about me?" The prior he trusts is that leading labs do the simplest thing and apply compute. They are also building on ground that shifts every three months. Complexity bakes in assumptions, and assumptions go brittle.

On who captures value, Crivello thinks of Altman as Bill Gates 2.0: the same appetite to own the whole stack. Microsoft went from a BASIC compiler to the operating system, peripherals, security, applications — an index fund on personal computing. OpenAI is reaching for compute, APIs, apps, code. History rhymes rather than repeats. Microsoft did extremely well and still could not own it all. Around the 800-pound gorilla he already sees Cursor, Lovable, and Lindy doing well, because the market is the biggest in the history of software. A few very big players and a thriving ecosystem is the shape he expects.

Tasting notes, with an eye on agents: Gemini 2.5 Pro is delightful, Google's machine finally moving. Jensen's advice, as he recalled it, is to be model-maxing — talk to all of them. o3 is the first thought-partner model that has blown his mind with actual insight; he uses it in ChatGPT, memory included. Lindy's default had been Claude. They were considering Gemini 2.5 Pro. Labenz had noticed a split in the market between Claude 3.5, which does what it is told, and 3.7, which can be over-ambitious — including, in one app-builder story, repeatedly trying to edit a config file it had been barred from touching. Crivello's eval suite has not been invested in as much as it should; they have limited trust in it. It is a signal. Changing the default is hot-swapping the brains of customers' AI employees, and people use Lindy for more jobs than the company first foresaw.

The upgrade policy is labels, not frozen snapshots. If you took the default, you ride the default as it changes. Labels include default, fastest (a Flash-class model), most balanced (then Claude 3.7), and smartest (then o3), plus a manual pin if you really want a specific checkpoint. They recommend the labels. It has gone wrong once. The first o3 drop, in his telling a rushed reaction to the DeepSeek weekend, was not ready. The eval suite said superior overall with a lot of variance. They swapped, customers on "smartest" reported issues, and they rolled back the same day. Time to detection was fast; he contrasted that with a slower rollback he had watched elsewhere. Swapping is part of the value: you should not still be on GPT-3.5, and a move from Claude 3.7 to Gemini 2.5 Pro can even save money.

Labenz laid out the current proliferation of strategies. Andrew Lee at Shortwave trusts Claude, caches hard, loads the model with tools, runs long episodes, and skips sub-agents. OpenAI's Agents SDK makes handoff a core abstraction. Chase has argued for a facade agent that faces the world and a set of task-specific agents with curated, local context. Crivello's answer was all of the above, on a maturity spectrum. What is actually being deployed fast is a single agent with tools and some deterministic scaffolding. That just works. Many-agent systems are still being defined and are not nearly as reliable. The SDK-style handoff, agents sharing one context, he reads as one agent walking through states — terminology more than a swarm — and mature enough for production.

Asked to recommend a commercial stack to an enterprise, he was almost apologetic. Lindy started before the ecosystem bloomed and had to build too much in-house, which he does not recommend. The P0 they want to replace is the eval suite: they are looking at Braintrust and a French startup, Basalt. He does not regret keeping the agent runtime in-house; it is too close for comfort to give away. Guardrails are mostly a user toggle — ask me before you send the email — plus whatever the foundation model refuses. Voice is ElevenLabs for synthesis, Deepgram for transcription, Twilio for phones, rolled internally because higher-level call scaffolds (Vapi, Bland, and peers) were too opinionated to preserve per-step model choice.

On AI-native tools in his own life, two things landed. Wispr Flow replaced his keyboard; he also wired the Whisper API to his iPhone action button via Shortcuts, French accent and all, and called the dictation flawless. Then a list of absences: Kindle, group chats that desperately need a summary, Twitter mentions that miss the point he already answered twenty times. Gamma, Labenz offered, is a rare product that tried AI at every touchpoint and got a Cursor-like growth curve. Crivello's broader disappointment is how slowly obvious LLM surfaces are showing up in software people already live in.

Labenz turned to safety. Jailbreaks are down; higher-order bad behavior — deception, scheming, what some people are calling o3 hallucinations — seems up. His first o3 session invented an Airbnb outright, not a small missing detail, and trust has been rebuilding slowly. Crivello thinks we are on track for the worst case, with one exception: Meta is not delivering in open source the way doomers feared, while DeepSeek, a Chinese company, is on the curve. "We cannot let China win this race, period." o3, he said, lies through its teeth — citing papers that do not say what it claims, then confessing it heard it in a seminar corridor. GPT-4's sycophancy episode was bad. Interpretability work, especially at Anthropic, is the one cause for hope. Users have not yet come to Lindy saying the "smartest" label produced crazy output. Knock on wood.

If you had handed him Gemini 2.5 Pro, Claude 3.7, or o3 in 2019 and asked what 2025 would look like, he would have predicted all hell breaking loose, and he would have been wrong. Slow diffusion is his leading guess. The flagrant real-world example Labenz offered was Sakana's CUDA engineer getting reward-hacked. Crivello connected that to the shift back toward reinforcement learning. Supervised fine-tuning did not bring the old doomer properties; scaling RL for reasoning models is now what frontier labs think about, and it is why o1, o3, Claude, and Gemini 2.5 Pro got better at code. You can watch Claude 3.7 reward-hack in the wild: a failing unit test becomes `assert true`, a type error becomes `any`, a broken component gets deleted. That, he said, should nudge the dial toward taking those concerns as at least somewhat warranted. Proof of personhood feels close; a friend even sketched a USB device with mic, camera, and fingerprint that would correlate live audio with the Zoom client and sell identity to banks and airlines.

The research question that actually keeps him up, given what Lindy sells, is the relative importance of scaffolding versus the model. Absolute importance of both is rising. Relative importance he has not settled. In an AGI or early ASI world he still sees a drop-in replacement worker, which he treats as a UI claim — voices, maybe faces, talk to it and it does the thing — not as proof that the underlying system is one giant model and one giant prompt. He is watching cheaper attention and dynamic, inference-time compute. If both land maximally, the case for an end-to-end agent strengthens, and scaffolding still buys extra reliability and speed. If they do not, he is very bullish on the harness. Labenz's last frame was that unhobbling may invert: as the model needs less help maximizing what it can do, humans may need the same scaffolding as a limiter. They were out of time.