Three Million Emails, Ten LLM Calls, Six Models: Shortwave's RAG Playbook

Nathan Labenz has been on Gmail for something like twenty years. He has tried other clients, inbox-zero rituals, and new interfaces; none of them stuck. Shortwave did. After a couple of weeks he was still opening it past the point of curiosity, into what he calls a genuinely new habit, which is no small thing given how deeply the Gmail muscle memory is entrenched.

The feature that sold him is the AI assistant: a natural-language sidebar that searches the inbox, checks the calendar, and drafts in his voice. Search alone would have been enough, he says, but the writing-as-me quality is the best he has seen in any app. As someone who gave up on inbox zero long ago, he just wants help navigating an overloaded mailbox — and he has had a few magical moments finding things he knew existed but could not keyword.

Andrew Lee, who founded Firebase, sold it to Google, and then started Shortwave, is unusually open about the stack underneath. There is no secret weapon. The assistant is a handful of models, on the order of ten LLM calls per question, and a retrieval pipeline broken into small, testable steps. Labenz brought him on *The Cognitive Revolution* for a long engineering walkthrough of that pipeline, from first import to the final GPT-4 Turbo call.

AI was not on the roadmap when Shortwave started. Lee had sold Firebase, worked at Google, taken time off, and gotten restless. Two things in 2019 pushed him to found another company. The first was the democracy protests in Hong Kong, and the way WeChat was used as a control tool: say the wrong thing and someone might show up at your house, or the message might never land. He told himself that at least Western messaging platforms were trustworthy — then started to doubt that too.

The second jolt was closer to home. Google killed Google Inbox, its own next-generation email product. Lee's reaction was blunt: if Google, with infinite resources and the largest email user base in the world, would not invest in figuring out the future of email, maybe he needed to. He had grown up running an ISP in the family basement with his father in the 1990s, and he still has a soft spot for the federated protocol underneath mail. He called old Firebase colleagues. They started at the beginning of 2020.

The first product was collaborative email, because Firebase had been a collaboration platform and they knew how to build real-time shared systems. It was hopeless as a v1. People shopping for an email client were not also trying to leave Slack, and people shopping for Slack were not about to pitch their whole company on a new mail client. They pivoted to a personal business client and spent the next year and a half on display, threading, and search — the unglamorous work of making email look right.

About a year and a half before this conversation, the client was solid enough. Large language models, which had never quite felt production-ready when they played with them, suddenly did. An old co-founder came in with a demo. They pivoted hard into AI. The realization that followed is the start of the whole RAG story: for the entire history of email clients, the data in the inbox had been inaccessible to the computer. You could search. You could read. You could not ask a question and have the client figure out the answer. Gigabytes of unstructured text went from a pile eating disk space to something that might actually be valuable.

Lee admits they got lucky on architecture. The collaboration product had required a client and a server, with real-time streams of data — the same shape as Firebase. When they turned to language models, that choice became the infrastructure. Unlike almost every competitor, Shortwave already had all of a user's mail on the server. Sign-up means a full import and reindex, which is expensive: they store a copy of everything. Some inboxes hold three, four, or five million messages. The job can take a few minutes or a few days. Users watch messages appear reverse-chronologically as the crawl walks backward.

Each message is indexed two ways. Elastic handles full-text search. An open-source embedding model, Instructor XL from Hugging Face, writes vectors into Pinecone. They picked Instructor XL because it was cheap, fast, good enough, and not too wide. With millions of stored emails and relatively infrequent queries, the vector database was going to dominate cost, so they did not want the biggest, best embedding. A medium-size vector is less precise; they compensate by returning more neighbors and stuffing more of them into the prompt.

Pinecone won on a feature the other top-tier stores lacked: namespaces, so a huge number of users can sit together without a performance penalty. PGVector, which would have been convenient because they already run Postgres, applies filters in the wrong order for this shape of data. It finds nearest neighbors first, then filters by user. Mix every tenant's vectors in one table and the engine sifts a mountain before it reaches your mail. Per-user tables are a workaround, with their own operational headaches. Lee's advice: decide how many namespaces you need — one per user, one per company, or one global — and be honest about document count and document size. Most RAG apps live in the thousands to hundreds of thousands. Shortwave does not.

Because they rerank with a cross-encoder, the vector search does not have to be razor-accurate. They cast a wide net, then score each document against the question. That trades retrieval-time latency against storage and embedding cost. Labenz notes OpenAI's newer embeddings with variable-length vectors, capturing most of the signal in a small size and adding resolution as you grow. Lee thinks that is the right idea: different apps sit at different peaks. He also expects vector databases to split compute from storage, because today's bundled pricing punishes a store-heavy, query-light workload like theirs. Pinecone's then-new hosted option was already on their list.

What impresses people about the assistant, Lee says, is that it works at all. Plenty of teams have demo agents; few have wired them into a product people actually use. Shortwave is not doing anything he would call revolutionary. "We're not AI researchers," he says. They are application developers who duct-taped the right components onto traditional infrastructure. The core insight came early: they could not get multi-step planning to the quality they needed. Each step fed the next, each model made errors, and the errors propagated. They flipped the design. The goal became one prompt that contained every piece of necessary context, answered in a single LLM call. All the clever engineering is about what goes into that prompt.

It still looks a lot like an agent. First comes tool selection: one LLM call, stuffed with world state — who the user is, the time of day, their settings, the thread on screen, the first few paragraphs, whatever is selected in the inbox. Context matters. If the open thread is about the Super Bowl and the user asks what time the game is, the system can resolve that; in the abstract it cannot. The call returns an array of data sources, six or seven tools. Some are cheap: dump the current thread, inject settings, or load extra prompt instructions for summaries. The expensive one is historical mail. When the UI says "searching," that is RAG kicking in.

If RAG is chosen, query reformulation turns a contextual, pronoun-heavy question into a standalone one — names instead of pronouns, absolute times instead of relative ones — using the thread history. Feature extraction, with a smaller, faster model, then pulls labels, contacts, time ranges, and keywords for traditional Elastic searches. Those run in parallel with a KNN lookup in Pinecone on the reformulated query. The two piles are merged. A hand-tuned heuristic layer sits on top; recency bias is the example Lee gives. If someone asks why their flight is delayed, it is probably the flight discussed recently, so older hits get marked down.

A cross-encoder running on their own GPUs then scores each remaining message: how much would this document help answer the question? Another heuristic pass, a sort, and the top roughly thirty to fifty threads go back to the client. The client makes the OpenAI call, with data from every selected tool — RAG is only one of them — plus formatting instructions so the answer can be parsed into rich text, thread links, and buttons. That last call is GPT-4 Turbo. Everything before it exists to make sure nothing important is missing from the prompt.

Labenz counted four language-model calls in the loop. Lee added a class he had missed: feature extraction. After the query is rewritten, several small models pull out labels, keywords, and other filters, each as its own call. Serial steps are five — tool selection, rewrite, features, cross-encoding, final answer — and feature extraction itself is about five things in parallel. Every time you ask the assistant a question, Shortwave makes on the order of ten LLM calls. That does not count the millions of embeddings written at signup.

Six models run in production. Three come from OpenAI: GPT-4 for the assistant's final answer, GPT-3.5, and a fine-tuned GPT-3.5. On their own infrastructure they run Instructor XL, a cross-encoder trained on data Microsoft open-sourced, and Mistral. GPT-4, GPT-3.5, the cross-encoder, and Instructor XL all serve retrieval. Mistral writes inbox summaries, and Lee has been impressed enough that they may put it in more places — it runs on their machines, which helps cost and latency. The fine-tuned GPT-3.5 is reserved for AI autocomplete, a Copilot-style inline experience built because people liked the writing but found the sidebar slow.

Autocomplete needed GPT-3.5's speed and could not have GPT-4's latency. Stock GPT-3.5 was not smart enough about the nuances of a completion: it would dump two paragraphs when a sentence was wanted, mangle whitespace, fail to sound like an email, or repeat everything already typed. Fine-tuning closed the gap. The dataset is the founders' own mail — no customer data. The recipe is simple and clever: take a real sent message, remove a section, and train the model to emit the missing piece. That taught formatting, typical length, and a subtler skill, copying facts out of context.

Lee's favorite demo is `our Wi-Fi password is:` — the model should paste the password from an earlier message. Without fine-tuning, GPT-3.5 could not see that even when the fact sat above. With it, it can. At training time a script walks their inboxes, assembles the full RAG prompt that would have been built in that situation, knocks out half a random sentence, and uses the actually sent email as ground truth. They also insert contrived examples for cases where the model misbehaved. What surprised him was generalization. He worried it would start rewriting their old mail. It did not. It learned the task, then at runtime respected each user's retrieved context.

Runtime retrieval for autocomplete is two queries, not one. When the cursor lands in the box, Shortwave finds sent mail in reply to similar threads — when the user got a message like this before, how did they respond? Separately it embeds the draft so far and finds similar sent mail, so a line like `my address is:` can pull the address. Both sets go into the prompt. The assistant stack is close but not identical: it uses GPT-4 Turbo because people will wait, and it can run arbitrary retrieval, not just mail you have sent. That is why Labenz finds sidebar drafts a notch higher quality. Labenz offered a tip from Waymark, where chain-of-thought in the fine-tune data was the unlock for their script writer. Lee had not tried that. He said they should.

Given the machinery, Labenz is struck that the product still feels fast. Lee's rules of thumb are blunt. Parallelize everything that can be parallelized. Pipeline everything that can be pipelined: cross-encoding starts as results leave the database, not after the full set arrives. Use faster models when you can — GPT-3.5, the fine-tune, Mistral. Put summaries on expensive GPUs because they have to be instant. Keep the output short. Chain-of-thought helps them too, but they use it only in sparing cases. The reasoning tokens come out before the answer, so the user waits. When performance matters, they take the reasoning hit rather than the latency hit.

A blog post from the previous October described the stack as cutting-edge. By the recording, Lee called the whole thing obsolete. Every stage is a candidate for replacement, even if the overall frame still works. Tool selection, which they could not fine-tune at first, might move to Mistral for cost and speed. Query rewrite and feature extraction might collapse into one smarter prompt. Better embedding models have landed; compute-storage split on the vector store might let them use a wider vector without blowing the bill. Reranking is the least settled. Traditional infrastructure work — filtering, plumbing — still has room. The goal is an assistant you can actually rely on every time, not one that is magical on some queries and empty on others.

Labenz likes the UI that already exists: a generation cites the threads it used, with a line separating used from merely retrieved, and the user can rerank by hand and fire again. He would show those threads earlier, and he wants first-class voice in, even at a desk, because forming the thought is faster than typing. Lee says they will probably do voice soon. Until two weeks before the taping there was no mobile assistant; on the phone his own two jobs are finding something in a hurry and turning garbled dictation into an email that sounds like him. Built-in iPhone dictation is not Whisper. Desktop is less urgent because a keyboard is sitting there.

Shortwave is losing money on every customer. The retail SaaS price does not cover the import, the copies, the embeddings, or the many model calls. That is not an accident. Lee's rule for a startup riding a trend is to launch a feature right before it seems possible. Impossibility can mean the technology does not work yet, the product behavior is unsolved, or the economics do not close. If you trust the cost curve, you cover the gap with venture capital until they do. YouTube is his example: it hemorrhaged money on bandwidth, storage, and re-encoding, and then the unit economics arrived. He wants Shortwave battle-hardened, with customers, just as the rest of the market decides this is a good idea.

They pitched the assistant as a cheap slice of what people already pay executive assistants to do. Labenz, who advises a company in that space, thinks the complementarity is real right now even if AI substituting for labor is the longer arc. A human EA dropped into a twenty-year-old cluttered inbox does not know the contacts; Shortwave's search is an unlock for her, not a replacement. Lee agrees the product will not do everything a person would, and they get requests for a proper delegated inbox so an assistant can work without sharing login credentials. People using Shortwave as their assistant today are usually stacking it on a human, or using it because they cannot afford one.

Labenz would happily delete about ninety percent of his mail, for Gmail storage overages as much as for retrieval quality. Lee would love to help. The number-one feature request is AI triage: today Shortwave writes and searches, and it can prioritize a multi-select, but it will not come in and throw away most of the pile. They are shipping safer, lower-risk jobs first. An agent that bulk-deletes mail needs a lot of confidence. Embeddings open a clustering path: pull the inbox and, instead of a wall of threads, show Tinder-style cards for rough personal categories — next week's podcast, junk from a marketing site, newsletters — with a recommended action on each. One thing users will notice in the current UI: the AI cannot actually do anything. It can recommend and put buttons in front of you. Sending, deleting, and archiving require a click. That is intentional. Lee wants people comfortable asking anything, knowing nothing leaves without them.

Longer term he wants a reframing. An inbox is a pile of to-dos that does not feel like it adds value. With language models it can be a knowledge base: everything going on at the business, everyone you have ever talked to, the SaaS notifications, the invites. Before this recording, the product could in principle have surfaced who Labenz was and what they had already discussed. Once that corpus helps you, it can help a teammate. If Labenz later emails one of Lee's co-founders, that co-founder's assistant might talk to Lee's assistant about what is safe to share. Privacy is the hard part. The weird, awesome version is a company-level model that knows what all the personal assistants know.

Labenz pushes on the new equilibrium: if everyone can send personalized outreach at 100x or 1,000x, inboxes fill with high-quality spam, and your mass-customization model talks to their mass-filtering model at a volume no human can track. The common joke is, if my AI writes the email and yours summarizes it, why not send the bullet points? Lee's experience so far is the opposite. The better they get at generation, the more the mail looks like what the user would have written anyway — same length, fewer mistakes, faster. The volume problem is real. He thinks the social graph will carry more of the filter: who introduced you, whether there is a prior relationship with the business. Content gets easier to engineer. Relationships do not.

Asked what Google will do, Lee speaks as an ex-Googler who had Inbox people on his team and who debriefed that group when Shortwave started. He respects the company. He also thinks a product with Gmail's legacy, user count, and conflicting internal goals cannot innovate rapidly at high quality, for organizational reasons more than technical ones. Gmail's most important job at Google, he says, is not helping you finish your work. It is getting you signed in so search can be associated with you. Workspace is a secondary anchor. The AI features look to him like a mandate from on high: plug something in, hit the OKRs. Embedding two and a half billion users' mail would be a massive expense, and the people who want to make that kind of bet would rather do it on Gemini than turn the Gmail supertanker. Google already has two incomplete attempts at AI mail search: a Duet AI preview that was not ready when he asked, and a Bard/Gemini integration that clearly came from the Bard team, not from Gmail.