Always Bet on the Models: How Shortwave's Andrew Lee Shipped Tasklet as Agents All the Way Down

Nathan Labenz welcomes Andrew Lee back for a third appearance, this time as founder and CEO of both Shortwave and a new product, Tasklet. Shortwave is still one of the few AI apps Labenz uses daily on laptop and phone. The occasion is Tasklet: an agent platform that is supposed to sit between a chatbot and a structured workflow, taking a job in plain English and then figuring out how to run it.

Lee says the second product started as a Shortwave feature. Earlier this year the team got good at hooking the email agent to other apps through MCP. Users began pulling data out of mail and pushing it into Notion, Asana, or HubSpot, then asked for the obvious next step: run that prompt every morning before they sat down. Once the work happens while you are away from the keyboard, a tight UI between agent and inbox stops making sense.

So they took a bigger swing. From late May and early June through the week of the interview, the company built a general-purpose automation agent rather than an email add-on. Labenz's pitch to first-time users is to skip the flowchart: say what you want, then iterate. The product has two layers that matter in practice. You chat with a long-lived main agent that owns the job, and you inspect the individual runs it has already completed.

Traditional automation, Lee says, is a workflow product: Zapier, n8n, OpenAI's AgentKit. That was the right engineering move a year or two ago, when models were smart but not that smart. The lesson of the last few years is to always bet on the model and to give it more agency over time. People stayed shy of fully agentic automation because they did not trust reliability. He thinks that transition has arrived.

The upside shows up the first time something breaks. A workflow with no branch for an error state just dies. An agent can work around the mess, handle nuance, and is much easier to set up. Some competitors use an agent only to *create* a workflow; the runtime is still a constrained graph. Tasklet uses an agent for setup *and* for execution. Labenz's cartoon of an agent is an LLM in a loop with tools. Lee's version of "bet on the models" is about who holds the steering wheel.

In a workflow, traditional software is in charge and LLM calls live inside boxes. Tasklet inverts that: the model makes the big decisions and software executes tools. There is still a lot of ordinary code. Lee groups the product work into three buckets. Connections hook the agent to anything on the internet, not a pre-enumerated tool list. Triggers fire encapsulated sub-agent runs with limited permissions. Team features — sharing, oversight, audit, cost control — are still nascent, aimed at business operations rather than personal chores.

Labenz had assumed a trade-off: more setup buys more reliability, and a looser agent is easier but flakier. Lee flips the frame. The real world is messy and workflow tools break all the time; an agent can solve many of those cases. With today's models, most business apps will still be somewhat less reliable than a carefully drawn graph. Give it six months, he says, and that will no longer be true. Predictions of that shape have been coming true for years.

Reliability is not only a property of the model. Tasklet already uses the LLM to infer type restrictions on dynamically created connections, then enforces those types in code. If a user really wants a flowchart for one phase, Lee thinks the model could write its own guardrails and have code enforce them. He has not built that yet. The future he is aiming at is not a permanent split between the quick unreliable path and the hard reliable one. It is better in basically all scenarios, not too far out — maybe Sonnet 4.5, maybe 4.7.

Has the eval debate moved him? "Still running on vibes," Lee says. In their market Anthropic's models are clearly best at what Tasklet does, so they cannot switch off Claude. When a new Anthropic model lands, customer pressure to ship it immediately is tremendous. If an eval suite disagreed, they would still roll the model and maybe tweak prompts. Formal evals that slowed the product would have real business costs. They started writing Tasklet in June; it has already evolved a lot, and users are blunt that they want the smartest thing above reliability, security, and even SOC 2.

Testing still happens. They run a pile of internal triggers, dogfood on the team, then widen. Shortwave uses opt-in experiments and watches retention; Tasklet will do the same. Head-to-head, many models look fine on a single question. Some OpenAI models may even win a one-shot Q&A. The product is LLM-call, tool-call, repeated for a long sequence. Anthropic handles that iteration better, and a small edge each turn compounds over a hundred steps. The rest of the agent-app market, he notes, is also heavily on Anthropic.

Labenz asks about METR's time-horizon graph, where Sonnet 4.5 came in a bit below GPT-5. Lee does not know why the numbers diverge. His cynical take is that some labs play the metrics harder. His practical evidence is pricing: if GPT-5 were as good at these loops, it would not cost less than half as much, and people would switch. Anthropic did not cut price when OpenAI did. They measure work in turns, not human-hours. A 50-turn cap, originally a cost fuse against infinite loops, is now hit constantly — especially on computer use, where every click can be a turn.

Sonnet 4.5 felt like a real step up for computer use on vibes. The blockers when it fails are cost and speed, not intelligence: screenshotting a desktop around is tremendously expensive, and users run out of quota or hit the turn limit. The team has tried Gemini as a cheaper computer-use sub-agent. Distilling a run back to the main agent loses intelligence, so the current strategy is still to spend on Sonnet. Haiku just shipped in the product; it is too early to judge on computer use, but it could matter for cost.

What kind of machine to give the model is the most active debate on the team. Today every agent has its own SQL database, a code-execution environment, and limited files. Optionally it can spin up a Linux VM with a browser and a filesystem. Windows support existed, then died: expensive, slow to wake, and almost nobody used Windows-only features outside the browser. They have considered Mac. The next goals are a shell, a real filesystem, and making those pieces talk to one another.

Claude Code surprised them with how much it can do outside programming once it has a terminal. Users already upload a video and ask the agent to transcode it; ffmpeg in a shell would just work. Today generated code cannot touch the agent's database or its connections. If filesystem, database, code, shell, and UI could call each other, Lee thinks the result would be tremendously powerful. Doing it well is hard. A user might have a hundred agents; a hundred always-on machines would be predictably ruinous. Specialized cloud startups are flooding in — he had just spoken with one called Blaxel.

The requirement is the illusion of a dedicated workstation that is powerful while in use and costs nothing for the milliseconds it sleeps, including storage. You do not want every agent storing a full copy of the OS. You also do not want the agent to corrupt its own config and then be unable to boot. Reliability of the machine, not just the model, is now a product problem. The rest of the requirements, he admits, are still uncharted. They are figuring it out a day at a time.

The original pitch was recurring tasks. The first version of Tasklet had a setup phase, then locked into running mode: you could design the automation or run it, not both. Users who had wired an email-triage agent into Notion and Asana then wanted that same agent to just write one email. The team's answer — make a new agent — meant redoing connections and prompts. They flipped the design. Any agent can wait on triggers *and* keep chatting. Once that shipped, people started naming them. It was no longer "my email-creation agent." It was Joe, the EA, who both runs the overnight jobs and helps when asked.

The product thesis became heavyweight, long-lived agents you might return to for years, versus the ChatGPT habit of a disposable thread. Almost every paying user has something automated and recurring. The majority of messages, though, are the other asks sent to those same agents. Companies have automated billing. Some replaced a human EA. Music venues manage calendars and keep stakeholders in the loop. A commodities firm watches real-world events and recommends trades. Lee uses it heavily for marketing: nearly all Shortwave and Tasklet content now goes through a Tasklet agent fed from his notes and spreadsheets, including the sponsored reads on this episode.

Labenz's own archive-paper agent is the two-tier idea in miniature. At the high level he pointed it at the Cognitive Revolution site instead of writing a bio, and it turned that into a standing prompt for each daily run. When he noticed runs were not reading memory — so the same papers came back — he told the main agent, not the run. It updated its standard operating procedure. That, he says, is starting to feel like a virtual employee: high-level feedback on the work, without rewriting the flowchart.

Labenz wants to talk to an individual run — one more step, a one-off rewrite. Lee says they used to make sub-agent chat easier. Sub-agents can still pause for human input, but it is rarely used, and twenty chat boxes make people unsure where to type. The destination is: talk only to the main agent. What is missing is enough information about recent runs, a project they call introspection — sub-agents for cost and reliability, with the parent appearing to know and control them.

They used to let users edit the intermediate instructions the main agent had generated. That created a split brain: you told the parent one thing, then patched the child's prompt, and even if you reported the edit, the parent could not reconcile the two as the conversation continued. People barely used the editor, so they removed it. Next they may stop pre-generating those instructions at all. Updating a standing prompt while also doing the user's live task is slow and error-prone. Just-in-time, out-of-band generation that always sees the full conversation avoids a UI that manufactures ambiguity.

Labenz notes that OpenAI's GPT-5 prompt guide warns hard against contradictory instructions, and that alignment-faking and other strange behavior often traces to incompatible goals at different layers. Lee has asked to personally review any pull request that touches the main prompt — he is not otherwise the CTO-as-reviewer. The thing he looks for is conflict. Several people can write locally sensible instructions; someone has to read the whole prompt and ask whether the model, taking it as one document, knows what to do. That failure happens a surprising amount of the time, even when people are careful.

A Tasklet-suggested question puts the token budget on the table: 200,000 tokens in the current context, with longer windows available at a higher price, still finite. If you keep asking the parent to pull the October 1 run, then October 5, you fall off the end immediately. Lee likes the phrase context engineering. Building an agent, as he now splits it, is a simple loop, a model, and then all the work of plumbing the right data in. "RAG," he thinks, has been used badly; it does not describe tool calling. Context engineering gives that plumbing enough weight.

An agent that has processed 100,000 emails cannot fit them in any reasonable budget. You still have to give the user the illusion that everything they said, and every email processed, is being considered. Methods include hiding tool results (Anthropic has been public about this), LLM compaction of old spans, and giving sub-agents a tailored slice of history rather than the whole chat. The north star is one big chat that never forgets. The system breaks because you cannot fit it, you cannot afford it if you could, and the model is not infinitely smart anyway. Dump every processed email into history and it still will not reason well about which ones it has already handled.

The first memory trick was almost cartoonishly simple: a JSON blob in the system prompt, edited by a tool call. It worked surprisingly well until it got large, at which point agents that had been fine for a week or two fell over. Mutable state in the system prompt is also terrible for caching. They moved to a SQL database because models are "way better than I am" at writing SQL. Tell the agent it has a database, plus instructions for how to use it, and it works. They are not done. Compaction will need an uncompact tool for the spans that suddenly matter again. They are not buying Mem0 or SuperMemory; long-running work is too core to outsource. Knowledge-graph schemes like HippoRAG are interesting, but internal state is only one source. The agent also has to search Gmail, Notion, and every other connected system, and those corpora are too big to ingest. The bet is the same as everywhere else: connect to things, and wait for the model to get better at using ordinary search. Cognition's specialized retrieval model is a fine idea for someone else. Shortwave's semantic stack still beats Tasklet at mail search; it is also a huge share of cost and engineering. Tasklet cannot build that for thousands of services. Lee is optimistic that Gmail will eventually ship a semantic search API, at which point Shortwave can throw some of that infrastructure away.

The advertised number is more than 3,000 business tools, plus any API and any MCP. Shortwave has about ten integrations; Tasklet started down the same path until Lee's co-founder plugged in Pipedream and suddenly the catalog was huge. MCP went in next. Then they wondered whether an LLM could call an arbitrary HTTP API if you described HTTP in general and scraped the vendor's docs. Direct API connections worked "shockingly well." A paying user stopped using Notion's official MCP and switched to Tasklet's fully generated API path because it was more reliable.

The strategic goal became ending the connector-count conversation: work with everything. Hand-rolled integrations for the important stuff, Pipedream for the long tail, provider MCP servers, user-built MCP, raw HTTP, and computer use for things with no API — LinkedIn being the canonical case of a company that does not want you in. All of that is presented as one connection object with the same permissioning, so the rest of the agent does not care how the wire works. Credentials are their own stack: an authenticated, stateful connection on the user, then per-agent grants, on the analogy that each agent is a different employee. The team roadmap is IT-admin-defined connections with audit logs, cost controls, and policy, so employees stop wiring random AI tools into production systems.

Lee was extremely bullish on MCP early in the year; it even helped inspire Tasklet. After Claude 4 they rechecked the prior. If tool definitions just mirror documented API calls, why not scrape the docs and call the endpoints? Before Claude 4, models were not quite smart enough end to end. Now they often are. MCP still wins on OAuth in some products — Notion's workspace-wide auth, for example. For a while Tasklet used Notion's MCP login and overrode the tool definitions. Direct API is now used far more often than MCP. "Smart" intent-level MCPs that wrap many calls are rare, and Lee is not optimistic they will last: if the model can generate those higher-level tools, hand-rolled specialist servers get obsoleted. For hidden-gem integrations he names Rocket Reach for finding emails, Perplexity's search API for heavy research, and a pattern of giving the agent its own Google Workspace account and name so it can send as itself. Human-in-the-loop is still half-built. The team's canonical example is daily DoorDash lunch: dietary rules, guests, cancelled restaurants, not everyone in the office. They have recommendations with links; a human still clicks. Sub-agents can email you; what he wants is a sub-agent that escalates to the parent, plus a mobile app with push, because email is a bad interrupt.

Picks and shovels: Pipedream has been great, Google Cloud runs the VMs, Anthropic is the model. They do not use LangChain; the agent code is their own TypeScript. On the coding side the team is heavy on Claude Code, most people use Cursor, some prefer Codex, some use both for different jobs. Claude Code is wired into CI. They use Sentry and Statsig. They are not tracking lines of code drafted by AI. Lee suspects it is already a majority of lines, though not 90 percent, and that this does not mean a majority of the thinking. Writing the lines is often not the hard part.

Ticket-to-pull-request works when the system is already architected for the change. Otherwise the model will hack in a fix that functions and quietly breaks the design. Static pages — pricing, release notes, terms of service — were one-shotted in Claude Code; Lee still decides the content. His co-founder, a faster engineer, will screenshot himself pasting Lee's request into Claude Code and mail it back. For functional testing they point Tasklet at Tasklet. Computer-use work on VM sleep and remote desktop was debugged by asking the agent to inspect its own machine. They have it invent tool tests against Notion. The most recursive demo: computer use opening the Tasklet site and editing the agent from the outside. Hall of mirrors, Labenz says. It is starting to get strange.

SOC 2 is a week-after-launch gap, not a philosophy. They care about data privacy and plan to do it soon. Lee still thinks the control question — can employees at Tasklet misuse your data — is obviously relevant. It is also not enough. You now care whether the *agent* does the right thing, and there is no standard for that. Security people he has asked say it is the wild west. The surprise: he expected customers to be terrified of a rogue agent with mail and Slack. They have had, he says, not a single complaint of that kind, against piles of ordinary bug reports. He attributes it to setting expectations in the open — the whole point of the product is agency — plus per-connection tool grants. He is interested in AI-agent insurance on the Airbnb model: an early house-trashing incident, then a big policy that made hosts willing to trust strangers. Medium-term, Tasklet wants to be the thing IT is most comfortable deploying.

Last time they spoke, Shortwave was losing money on users until caching flipped the economics. Lee's update: several launches felt like they might bankrupt the company if usage exploded without revenue. That never quite happened. Monetization improved, costs fell, and more caching work helped a lot. Shortwave is now on healthy margins — not classic 90 percent SaaS, but good money. Tasklet is back where Shortwave started. It is all AI, people burn far more tokens per person, and the product is strongly margin-negative. He expects the same arc: years of red, then neutral, then strongly positive. Haiku 4.5 is a huge unlock, a third the price of Sonnet and good enough for most jobs; they were rolling it out the day of the interview, which should raise the daily quota without blowing the fuse.

The key to caching is an immutable log. Never modify earlier messages; every edit invalidates the prefix. Anthropic offers no cache, five-minute, or one-hour breakpoints, and you have to choose per slice. Shortwave's hit rate is around 85 percent. A big unlock was moving mutable state *out* of the system prompt and into system messages — a hidden XML-ish block prepended to the user turn. Put the current time in the system prompt and every minute nukes the cache, including across users. Omit it and the model does not know what time it is. A tool call for the clock costs a whole extra turn. Anthropic itself now uses a similar pattern for context-usage notices. Same trick for user state and Shortwave's memory. Compaction is the other large win.

A week after launch they are already staring at zombie jobs. A significant fraction of cost is free users with recurring runs. Some of those people may convert; some are emailing a mailbox nobody reads, at about a dollar a day in model spend. Paid users can keep paying. Free users who go dark will probably get an email, then a shutoff. Lee does not have a clean answer yet. Labenz's summary is ungentle: if you let free users set up cron jobs, you need limits or you will hand the company to Anthropic.

Labenz thinks the multi-agent future is under-theorized. Inside a single application serving one user, Lee has yet to see a good case for more than one agent. A big model, as much context as you can feed it, reasoning over the whole problem, still gives the best answers. Multi-agent starts to matter when different parties should not share data — two sides of a trade, an adversarial dynamic. That is outside Tasklet's scope. He shouts out Kradle AI, a company from his Firebase co-founder James Tamplin, which drops models into games like Minecraft and watches collaborative and competitive dynamics. On METR-style task length, Lee expects the curve to keep rising. Gemini 3 already looks incredible and will feel dated in three months. Blackmail demos he files as fun research: conflicting instructions in a eval harness produce unsavory escapes, which is not that different from how a human behaves. Whether a robot should jaywalk is a values question, not only a safety one.

Why not route marketing to a marketing agent and HR to an HR agent, the way n8n template sellers do? Foundation models, he says, are becoming best at everything. Two agents working for you differ only in context and tools. If one model can hold both, giving it both sets works better. That has a larger implication: most vertical SaaS goes away. Horizontal AI platforms that can dynamically generate UIs — they were prototyping this the morning of the recording — plus vertically integrated businesses that own an end-to-end outcome. Intermediate SaaS for construction or medicine gets washed out. Labenz's mantra is "AI beats UI." Lee asks how long until you can tell an agent, "Give me an email inbox designed for fast triage," at which point Shortwave ceases to be a valuable product. Not quarters, he thinks. Years. He would not bet on Shortwave in its current form in twenty years. A separate quality-control agent with a different prompt has not helped when they tried it.

They have avoided "virtual employee" in marketing because too many companies say it and deliver nothing like it. What sets Tasklet apart, in paying customers' mouths, is agents all the way down — it does not turn into a flowchart — and the fact that it actually works. Launch went well. They are adding Tasklet revenue faster than they ever added Shortwave revenue, by a huge margin. The work left is to make it smarter, more reliable, faster, simpler: from finding spam and generating invoices toward running a small restaurant's entire marketing function. Ethan Mollick's "best available human" bar is the near-term target for small-business work you cannot hire well for. Speed is still the only moat. Connector libraries used to take years; direct API plus computer use meant Tasklet had, on day one, better coverage than Zapier. An AI email client used to require first building an email client; before long you will just ask for one. By the time a listener of this episode has copied them, they intend to have moved again.