From Email Client to AI Agent: How Shortwave Rebuilt Itself for the Tidal Wave

Host Nathan Labenz opens by noting that Lee's recently posted revenue chart looks like a textbook exponential curve. Lee says the cause is a year of evolution in Shortwave's AI assistant. A year earlier you could chat with it and it did an okay job at search and drafting, but it could not be trusted and could not do the normal work of an email client such as managing to-dos, labels or contacts.

Now, Lee says, people can do their email by talking to the assistant, which behaves like a virtual assistant and can do almost anything a human user can. Email is a huge corpus covering correspondence, SaaS notifications, attachments and calendar invites, so a capable agent can give real advice. Lee used it to count how many users had complained about a new layout in the previous 24 hours, and got a figure of 19 along with the top five reasons.

Users have found their own uses. Many begin the day with a long prompt that triages the inbox and prepares for sales calls, while real estate agents and contractors ask questions about PDF attachments. One person selling a house pulled an inventory of furniture from years of receipts, and Labenz had it gather trip receipts into an itemized expense report, saving him roughly half an hour.

The most inventive examples came from users rather than the team. One person had the model generate clickable URLs that create tasks in Linear, which works because LLMs know the URL structure and needs no integration code. Another uploaded a list of addresses and had the assistant loop through them, searching past correspondence to write a custom email for each person for review before sending.

Lee says that between his two appearances nearly every layer of the stack was rewritten: the embedding model, the vector database, the search stack and its API, the agent models and the agent code. Each time a model improved, the team found that restructuring the system around it unlocked something extra. The pace of change, he says, is phenomenal.

The key shift is that an agent can run several searches, in parallel or in sequence, and adapt based on what it finds. That means the search layer no longer has to find the right email in a single shot, and it could be narrowed to a faster, cheaper and more reliable job. Shortwave moved to Pinecone's serverless offering, which separates storage from compute and suits a corpus as large as email, and now uses a bigger BGE embedding model.

The old pipeline combined smaller LLM calls, feature extraction and a reranking step, and Lee calls it slow and brittle. In its place the search API takes a semantic component plus ordinary constraints such as date range, contact or label. The system runs an embedding search and a keyword search, combines their scores and then post-filters, which Lee says is cheaper, faster and more accurate, though not perfectly exhaustive.

Lee ties this to a theme from his earlier work: improvement in one layer lets you delete cruft in another. He compares it to the contortions once needed to pick an image for a user, which a model can now do directly. Labenz adds that he watched the assistant retry several searches to find his contact-lens prescription email, just as he would have.

Lee says the assistant went from a single LLM call wrapped in a complicated rules-based machine to a loop in which a big model runs repeatedly until it gets the answer. The history is instructive: OpenAI's tool calling in GPT-4 two Decembers earlier seemed to make the model reason worse, and GPT-4o worked only partly. A September V2 agent that ran multiple searches brought some growth but was not dramatic.

A podcast with the Bolt.new founders led Lee to try the October version of Claude 3.5 Sonnet for tool calling, and it was dramatically better. Earlier models went off the rails after a few iterations, whereas this one could keep running many tools and still reason. Shortwave rewrote everything around a fairly generic agent framework, many more tools and a good prompt, and launched it in January as the V3 agent.

Lee says that when people talk about working agents, what they really mean is iteration. The model tries something, and if a search is malformed or a calendar invite conflicts, the system returns the error and the model tries again. Over many calls this reaches answers no single call could produce, and Lee points to that V3 launch as the kink in the growth curve.

Asked how a model knows when it has found enough, Lee says models already know roughly what the right thing looks like. Shortwave's inbox organizer spots cold sales emails with little prompting. The team also gives the agent tools: a contacts tool that returns statistics on how important a contact is, plus access to email history and calendar events. Lee admits the product does not yet learn from the triage actions users accept or reject, so personalization today requires telling it to remember something explicitly.

On model choice, Shortwave uses several models for different features: a fine-tuned GPT-4o mini for autocomplete, Llama 3.2 3B for quick replies, and a few GPT-4o calls. For the main assistant, though, Lee says splitting work across cheaper models added complexity and hurt the model's ability to reason across tasks. The savings did not seem worth it.

Anthropic's prompt caching is critical, Lee says. An agent re-sends a history that can reach hundreds of thousands of tokens, and if the earlier parts stay immutable, caching can cut cost by roughly 90%. It is harder to use than OpenAI's automatic caching, which offers only about half off, and Lee says it was one reason Shortwave moved to Claude. Without it, he says, each user would lose the company money.

Lee adds that the cache is built by checkpointing at each iteration of the agent flow, and an inbox organization run of about 20 tool calls adds up quickly. He mentions that when he spoke with OpenAI's agent SDK team the day before, caching was the top question from the other founders in the room.

Labenz asks whether the frontier labs are converging. Lee says the API surface has converged enough that swapping providers is easy, with aligned tool calling, growing agreement on caching and MCP becoming a standard. Beneath that, he sees Anthropic focused on iterative agent work and OpenAI more focused on reasoning and multimodality.

Shortwave uses models from three vendors precisely because each is strong at something different. OpenAI's serving infrastructure is fast and reliable, so latency-sensitive autocomplete runs there, while Anthropic's API goes down more and has a slower time to first byte. Cheap, latency-sensitive features such as summaries run on open-source Llama served from Google Cloud.

Lee's main advice to would-be agent builders is to retry what failed before October of the prior year, because the models and caching economics now make it work. He points to Bolt.new and Cursor's agent mode as examples. The harder open problem is UX: people understood autocomplete, but they will need new ways to oversee an agent that acts for them without being interrupted constantly.

Labenz asks about AI filters. Lee explains that this is the only Shortwave feature where the AI acts with no approval step: the user writes a prompt and chooses to label or delete matching mail. It is popular, there have been few losses, and each email is sent to OpenAI's GPT-4o mini, a vendor Lee says Shortwave trusts and whose terms prevent training on the data.

The goal is to run the full agent on each incoming email, so a rule could say to accept a meeting request from an investor who has already exchanged several emails with you. Lee names two obstacles. One is cost: an agent run happens now a handful of times a day per user, and running it on every email could mean hundreds of runs. The other is trust, including the risk that a malicious email could use prompt injection to try to delete an inbox.

Possible mitigations include limiting what the agent can do, for instance letting it touch only the email it is reading, and showing a history of actions the user can approve, from which the system could learn. Another is queuing actions instead of executing them, with drafts as the best example. Lee says the product must be powerful, trustworthy and resistant to people messing with it.

Lee says the team constantly tests new models, including Gemini Flash, but cannot keep up with everything, noting that a new release like DeepSeek triggers questions within days. Choices weigh cost, latency, caching behavior and task performance. Flash might first land in summaries and quick replies.

Evals are deliberately light because the company chose to adapt quickly rather than avoid breakage. Lee keeps a Notion document of about 100 golden prompts and runs the relevant ones by hand in his own inbox when changing prompts. He does not lock behavior down, since improvements such as a new unsubscribe tool that made the organizer offer to unsubscribe from junk would otherwise count as failures.

The more objective measure is an experiments framework: major changes ship as opt-in features to a user base of forward-thinking early adopters, and the team watches retention. The unsubscribe feature held about 99% retention after a week, which gave the team the confidence to roll it out. On provider outages, Lee says Shortwave has simply waited rather than failing over, and has not lost users because of it.

Lee frames the trade-off plainly: anyone who wants the most stable, unchanging product should use Gmail, and anyone who wants the cutting edge in an email client should use Shortwave. Labenz agrees that depending on imperfect services is the new normal and worth the occasional risk for the high-value moments.

Labenz notes that the new to-do categories created by the assistant do not sync back to Gmail. Lee says this reflects a conceptual shift: Shortwave used to be an email client with AI built in, and now sees itself as an AI with email features. That is why the assistant moved to the left of the screen, and why the plan is to integrate with other inboxes such as Slack or LinkedIn, plus CRMs and project tools.

In that world, parity with Gmail makes less sense. Labels are for short, simple long-term classification, while to-dos are short-lived, can have sentence-long names, notes and attached threads, and let the AI group related threads and documents for a project. Lee also says Shortwave avoids polluting Gmail, contrasting it with Superhuman, whose reminders are implemented as emails that remain clutter after you leave, so users can switch back to Gmail at any time.

Labenz says he has tried to fine-tune a model to write as him for two years and that nothing has beaten dumping writing samples into Claude. He also notes that no single channel holds all of him, since his email, texts, Slack and Twitter DMs capture different facets. A unified assistant spanning them excites him, and he reports that routine drafts, such as scheduling replies, are increasingly ones he sends as written.

Lee strongly believes that putting examples into Claude's context is the right solution, even though it is less technologically glamorous. His first reason is that this approach recalls specific facts better than a shallow fine-tune, and in email the correct link or meeting time matters more than a similar-sounding one. His second is that fine-tuning slows model upgrades, since every switch would require re-tuning everyone, and Shortwave switches models constantly.

Lee is more optimistic about reinforcement fine-tuning done generically across all users, to teach models how to use examples to match style and get facts right. On memory, he says Shortwave has a Memories feature where the model manages a list of facts that are inserted into the prompt, but users must ask explicitly. He expects some breakthrough in lightweight customization, and Labenz points to the Titans paper, state space models and HippoRAG as interesting directions.

On AI-generated spam, Lee has not seen a notable rise, and many of his own emails are partly or fully AI-written without recipients noticing. Labenz admits he may be too precious about his style. Both agree that routine emails are where AI helps most.

Lee says Shortwave raised more money, whose size he did not disclose, and has rethought how it is run. The company decided its future is an agent that does work across communication apps, and he sees competition as the next ChatGPT rather than the next Gmail. The biggest and fastest-growing plan is the most expensive one, whose users send more spending to Anthropic because they care about the AI.

Lee argues that every role is becoming one of managing AI interns. A good engineer should always have an agent running, and bug reports can sometimes be pasted into Cursor's agent mode and fixed in one shot. Designers should start from a working prototype built with Bolt.new rather than a mock, and one content person should be able to run the blog, social media, changelog and docs. The team let go a couple of talented people who were not passionate about the new direction.

Lee says engineers are significantly more productive than they were six months earlier, and prototype-first design skips a lot of bad iterations. He also describes a shift in how he answers investors about moats: building an email client took Shortwave four years, but the next entrant will be much faster, so code is a weakening moat. The only one that likely matters is speed, and small teams reach consensus faster. He cites Cursor and Midjourney as companies of about 20 people.

Lee says Shortwave is no longer losing money on each user and is modestly margin positive, though a large share of each dollar still goes to LLMs and email infrastructure. Growth is concentrated in the top plan because it offers the largest context window and indexes the user's full history. He thinks the company undershot on price, would pay more than ChatGPT Pro's $200 a month himself, and sees unlimited room for spending compute on a user's behalf, such as running a full agent on every email or using reasoning models for deep analysis.

Asked about architecting agent systems, Lee says Shortwave could become a routing layer for messages across a business, a Zapier-like tool with a human interface. Inside the product, however, his team tried multi-agent designs that switch prompts or models by problem type and found them brittle and weak at reasoning across tasks. A win came from putting custom instructions into the main prompt on every call, so an instruction like always addressing someone a certain way applies everywhere.

Lee predicts that OpenAI's new agent handoffs will not take off, and says he told the OpenAI team so. Labenz suggests that large companies' org charts may drive such designs, citing a remark that late-fusion vision-language models reflected separate teams. Lee agrees and says Google's Gmail team could not move quickly for organizational reasons, whereas moving the AI sidebar to the left took Shortwave only a few weeks.

Labenz asks why roles are in person in San Francisco. Lee says speed is paramount and that changing course and delivering tough feedback is emotionally harder remotely, and he and his co-founder work better in person. He recalls that Firebase was fully in person and that some of Shortwave's struggles came from being remote. He acknowledges that this limits the talent pool. An AI scout role is not on the list, but he thinks it would help.

Postings now include entry-level roles such as customer success, a content creator for videos and a junior product engineer. The application is a five-minute video showing something impressive the candidate did with AI, because take-home tests are too easy to game. Lee wants people who are forward-thinking and creative with the tools.

On software jobs, Lee says his CTO co-founder swings between feeling obsolete and feeling like a god. He thinks localized code such as front-end buttons will increasingly be written by AI, but understanding user problems and how components interact will remain valuable, and that is what makes a senior engineer senior. He expects an explosion of small companies and is an optimist, while Labenz worries that even 10 times more software might not sustain today's headcount.

Looking ahead, Lee wants post-training focused on agentic behavior and tool calling, with more competition there, plus cheaper and faster production tools and natively multimodal voice. He closes with a $10,000 referral bonus for anyone who refers a hire.