Seventy Percent of the Inbox Is a Pitch: Matt Welsh on the First Wave of AI Agents

Matt Welsh is the founder and CEO of Fixie.ai, a platform for building natural-language agents that connect to data, call APIs, and take on process work. The company had just announced a $17 million seed round when he sat down with Nathan Labenz on *The Cognitive Revolution* — a number Labenz joked had a little of the 2021 fundraising vibe. Welsh’s résumé is unusual even by Silicon Valley standards: Harvard computer-science professor, a decade as a principal engineer at Google, a stint at Apple, then this startup. Fixie was in developer preview, with a chat interface where you tag an agent the way you would tag a colleague in Slack.

The industry’s attention had only just swung to agents. Welsh had been building them for a little over a year, citing Google’s January 2022 LaMDA paper as the spark — a demonstration that the right prompts could teach a language model to call an external tool and drive a process. Labenz’s own wager was blunt. Cognitive work is already useful, but as long as humans have to figure out how to route information to and from the model, automation stays bottlenecked by implementation.

Agents, in that telling, promise to handle not only the reasoning but the messy details: searching, sorting, signing in, reading documentation, using APIs. Labenz sees no deep reason it should fail; reinforcement learning and a growing stack of tools look well suited to the remaining problems. He expected visible progress by the end of 2023. What he did not claim to know is how the world would respond once agents started talking to humans, to existing software, and to each other.

Asked what the last couple of months had felt like, Welsh had one word: bananas. He treats a system like ChatGPT as alien technology that just landed in the backyard, with everyone picking it apart to see what it can do. On the fundraising, his metaphor is that ChatGPT “chummed the water.” Investor attention flooded the space, and Fixie happened to be standing in the right place because the team had already been working there before the consumer moment arrived.

They were not waiting for InstructGPT. The work started in the early GPT-3 days, when almost nobody was building companies on the API — Jasper.ai being one of the few — and almost nobody was using the model to drive automations or reach custom data and external tools. Open-source projects such as LangChain were just starting to show demos. The paper that actually moved Welsh was LaMDA: give a language model the right prompts, teach it to fetch external data, and you can run an interesting process. There were barely any papers on that topic at the time.

Yes, the models improved from GPT-3 to GPT-3.5. What changed everything, he argues, was popularization. The same work moved from a clunky playground buried behind OpenAI’s developer site into a consumer-focused chat box. You could have done most of ChatGPT in the GPT-3 playground, he says, “just nowhere near as elegantly and usefully.” Capability was already on the table. The team that dialed in the end-user experience detonated the market.

Labenz had been splitting AI usage into copilot mode and delegation mode. In copilot mode you are still driving: the model drafts, answers, or runs a little analysis at your beck and call. Delegation is carving off a whole job and sending it away — not architecting the system yourself, but asking the model to figure out which buttons to push. He had started to hear that second pole as process automation, with agents as the bridge that lets a human stay in the pilot’s seat while handing off distinct pieces of work.

Welsh upgrades the sketch into a plane. One axis is whether the system runs in line with a human or as fire-and-forget autonomy. The other is whether it interprets and synthesizes natural language or automates a process. Early applications clustered on language-in, language-out. Jasper is the clean example: you describe a product or a blog post, and it writes marketing copy. That is the copilot-plus-language quadrant.

On the automation end, you start treating the language model as a computer you program in English — one that can do rich symbolic work not only on human prose but on a JSON object, a spreadsheet, or an XML document. Almost any point on that plane has a use case. Fixie is aiming at the opposite corner from Jasper: more autonomy, more process, and not necessarily a live conversation. Labenz called it one of the best questions and answers the show had had. He would later add a third axis that keeps coming back: how long the agent lives, from a one-shot search to a companion that remembers a whole day.

Unemployment is the elephant in the room, and Welsh does not dodge it. Most of the time, he does not think AI automation puts people out of work; in some cases, jobs done by hand will become obsolete. The black-and-white photograph he reaches for is a room full of accountants hunched over ledgers and mechanical calculators, doing in a day what a single spreadsheet does now. The question is which tasks look like that ledger, and how soon.

Labenz is advising Athena, which hires executive assistants in the Philippines — about a thousand active EAs, working one-on-one with clients, mostly in the United States. Internally they call email ghostwriting “pre-drafting,” and they have built training around it. His mission statement for the company is to become the EA shop that is best in the world at adopting AI tools: “please do my job for me.” Welsh uses an assistant through a company called Double and sees the same split. One side is written content — replies, scheduling requests, a newsletter — where non-native English speakers in particular have a lot of headroom. The other side is mechanical process.

The concrete example is his own inbox. As a CEO, roughly 70 percent of the email he gets is people trying to sell him consulting, recruiting, a new bank account, something he does not need. He has found it easier to spend a minute on a polite no than to ignore the message and absorb five, ten, twenty follow-ups from the sender’s automation. The catch is that you cannot tell whether a message belongs in that pile until you sit down and read it. Today his EA does that sorting. It is, he says, a laborious process that AI should already be able to take on.

The models are capable enough. What they lack are appendages into the systems where the work lives: the inbox, the calendar, the APIs. Labenz adds a user-side catch from his own time as an Athena client. Routine calendaring delegated cleanly; email drafts never quite got there. Nothing was wrong with the text, it just was not how he would have said it. That last mile of voice is a gap people and models will both have to cross.

Labenz had already been on the Fixie preview. He was comfortable granting the official calendar agent access to his calendar. He would not authorize a Gmail assistant built by a hackathon user. That is the trust model in miniature: people keep a mental ledger per agent and per permission. Welsh is explicit that the current preview is a free-for-all, useful for surfacing use cases and the wrong shape for an enterprise. Companies will not plug a natural-language agent some stranger put on the internet into the intranet. Most will build their own integrations, because they need to know where the data flows. The judgment itself is not new. People already make it for Zoom, Google Calendar, and the Slack apps that can see a Drive.

The new problem is the path of a single question. Imagine an agent that slurps an internal Notion workspace full of proprietary pages, then answers “what is the latest on this project?” The query has to route to a language model — which model, hosted by whom, who else can see it. It hits a vector database for relevant snippets — who runs that, where does it live, how do you delete it. The conversation history lands somewhere too. Fixie primarily calls OpenAI, with Anthropic as an option; an external store such as Pinecone is another vendor in the software supply chain. The biggest gap, Welsh says, is a model good enough to host behind the firewall. Azure’s OpenAI service helps some buyers. The week they recorded, RedPajama and Stability AI had each just shipped another open-source project, and he expects an onslaught that makes private deployment real.

Then there is the model itself as an attack surface. Prompt injection is not solved; people will hide instructions in HTML the way they once stuffed pages to fool search engines, and “the internet remains undefeated.” Welsh’s first take is that a lot of value is available even when confidence in absolute safety is low, because many jobs already have a human in the loop. A draft email that a person reads before sending is a natural speed bump. His second take is software around the model. He points to Guardrails, an open-source project from former Apple engineer Shreya Rajpal, where you write a spec — a weather agent must talk about weather, temperatures must be in a sane range — and check each response against it. It is not a panacea. The further idea is to train small, cheap models whose only job is to score whether another model’s output meets a spec. Labenz hears echoes of Eric Drexler’s comprehensive AI services, Emad Mostaque’s “million small models,” Constitutional AI, and even a guess that GPT-4 might already be doing a kind of semantic validation inside.

Memory is the other live wire. Fixie uses a session: an append-only conversational log between a user and any number of agents. Ask for a pie chart of the top ten committers on a GitHub repo, then say “make it a bar graph,” and the session supplies the rest. Sessions are mostly ephemeral today; a permanent session is technically easy, and it is roughly what AutoGPT and BabyAGI do with history. Most enterprise cases do not need that. Win a single customer-service call and you are already winning. The privacy failure mode is the shoe ad you clicked on CNN that follows you for three weeks. Permanent memory can be poisoned by something irrelevant, and the agent veers off. Welsh’s homespun fix is two Chrome profiles, work and personal, never the twain shall meet. On the platform, memory is explicit — an agent stashes a fact on purpose — and it has to be segmented by end user, so a support agent never mentions B’s order while handling A’s ticket.

What does sitting in front of a computer look like in six months, or two years? Welsh points at two early products: Ember, a natural-language agent embedded in the desktop, and MultiOn, embedded in the browser. The qualitative shift is when these things stop waiting for a hotkey and start watching everything that happens. He jokes that his MacBook Air *is* Fixie — email, Zoom, documents, code, the whole company in one small package. There is no reason an agent cannot have the full state of his work life, and most of his personal life too. Siri reminders are a toy next to a system that already knows what you have to deal with.

Three pieces have to come together. First, a streamlined user experience; ChatGPT proved the point, and so did the web. Hypertext existed before the web, he notes, in a system called Gopher, and it was awful. Conceptually the same thing, presented badly. Second, a clearinghouse for personal information you are willing to share. Browser history, calendar, email, texts, and Twitter currently live in separate silos, and an agent that is supposed to help needs the same access you have. Third, transparency and control: users have to see the grant, trust it, and revoke it. Ideally, he says, that stack is personal — running on your own device — because people will not be comfortable handing every digital act to Google or Facebook, even if they already do so without quite noticing.

He used to be on the Chrome team. Chrome, by default, can keep tabs on every site you visit and every link you click; most people never look at the buried history tab, so they never feel the surveillance. Labenz floats a half-baked idea he cannot shake: bring your own embeddings, so apps get a semantic index and the raw data never leaves the device. Welsh likes it and hears a cousin of federated learning. Embeddings can be inverted fairly well today, though there is research on one-way transforms that still allow nearest-neighbor search. He does not believe the world will wait. The early web had no HTTPS; everything ran in cleartext for years, and even after SSL arrived, sites were slow to adopt it. People will happily grant the Facebooks and Googles of the future access to sensitive data if they think they are getting value.

Apple, where Welsh also worked, is the counter-bet: personal data should live on the device, processing should happen there, and the company is not trying to mine that data for ads. It holds them back on targeting. He still thinks it is where the world ends up, especially once the chips make on-device inference real. As for form factor, he is not reaching for Apple Glass. He thinks it looks more like *Her*: an earpiece you can talk to, with the phone as the screen you already carry. At Google he was one of the few people who dogfooded Glass in the real world; it never caught on, largely because a camera on your face was not socially acceptable. Earbuds hide. Documents and large screens will still want text. Out in the world, earpiece plus phone is the thing he expects to take off.

Labenz closes on hopes and fears. Welsh’s hope is almost earnest. Today, taking full advantage of a computer still means joining “an elite priesthood that knows how to write code,” or at least how to bend complex software into shape. Digital-literacy campaigns have tried to make machines easier. AI is not just a friendlier interface; it brings reasoning and task execution with it. Put that in everyone’s hands, and you no longer have a world where only the privileged, the wealthy, and the well-educated can use the machines. He thinks that is a second information revolution.

The fear is closer to home. He thinks about his kids, and about a world where the old ways of learning and of being graded fall apart. If you are no longer evaluated on how well you write an essay or how clever a program you can produce for an assignment, then what is the curriculum, and who gets the A’s, B’s, and C’s? Those structures, he notes, were never a clean measure of intellect anyway. They measured other properties that happened to be convenient to score. With AI in the mix, the rules of how the world is organized have to be rewritten, and nobody knows what the new ones are. His own stance is cautious optimism, with a real unease about what that rewrite will look like.