Notion's Simon Last on Rewriting the AI Harness Every Six Months and Managing a Swarm of Agents

Last says his interest in AI had been growing for a while, but it became real in 2022 when he and co-founder Ivan got access to GPT-4. They saw two things at once: it was smart enough to follow reasonably complicated instructions and write or edit text, and its world knowledge was broad and deep. It was immediately clear to both of them that the time to apply it was now.

They started with a short-term and a long-term plan. The short-term idea was a writing assistant inside the document, which a tiger team launched in two or three months, in February 2023, as AI Writer. The long-term idea was a general assistant with the same tools a human has in Notion, able to create and query databases and weave them into longer tasks. That one did not work yet and took much longer.

The next big step was Q&A: a semantic index of the whole workspace so that answers are grounded in sources. Its general availability came in October 2023, and it was a far bigger effort because it needed a real-time updating index and serious evals. Once the Notion index worked, the team added Slack, Google Drive and more sources on a regular cadence.

The host asks what right Notion has to index tools that natively struggle with their own search. Last admits that question comes up often but says most companies are bad at their indexes. His view is that it takes some AI savviness and then a lot of craft: each source is different, so you must try real queries, use the product daily and keep tuning.

Asked how Notion copes with the messy variety of real workspaces, Last says embeddings have largely changed the picture. The AI does not care about the tree structure, only that a snippet with the needed context can be retrieved. Notion now advises users not to worry about organization and just pipe everything in, while chunking strategy and pipeline steps remain important but are invisible to users.

Asked how often the team has rebuilt things, Last says the AI harness is rewritten about every six months, and the interval has been shrinking. He thinks many companies get this wrong by building once and sticking with it. You have to stay keenly aware of the state of the models and design the harness around it, and he finds the restarts fun.

A new harness version was due within a week or two of the interview, and the team was already thinking about the one after it. Coding agents make that willingness to rewrite stronger, since the agents can help do it.

Last describes eras of coding assistance: tab autocomplete, then inserting and rewriting code, and then actual agents. He began using Claude Code around April of the previous year, which he calls a huge unlock. Agents can implement, verify and maintain features end to end, but that demands careful thought about architecture and the verification loop. Done well, you can build more ambitiously and more robustly than humans alone, and done badly it is all slop.

The fundamental effect is that individual output rises, and it increasingly depends on the willingness to use the tools. Notion has not seen much change in team size, since it already prefers small tiger teams. He says the gap between engineers is bigger: the minimum bar is the same but the maximum has increased enormously, so a 100x or 1,000x engineer is possible.

Internally things feel messier and more chaotic, which he likes. There are many more prototypes, such as a design team repository called the design playground that lets designers deploy high-fidelity prototypes at a URL instead of pointing at mocks. Every pull request is now agent-written, often larger and more complex, but also much better tested, and all still go through review.

He says he never produces a PR that has not been fully tested end to end. It is not pure vibe coding: you think carefully about the change, how to verify it and how to deploy it safely, and enlist the agent to help.

Last says the team struggled for a few years to build a real agent. They tried three or four times, but it was largely too early, and the version that finally launched came in late summer or fall of the previous year. Now every Notion user has a personal agent with access to what the user can access, able to create databases, update things, write documents, search the web and do research.

The second launch, a week before the interview, was custom agents. Unlike personal agents, they have no access by default, and you grant it. They can then run autonomously in the background, for example attached to a Slack channel and a task database to answer people and fulfill tasks, or reading a database of weekly reports while searching the web.

What excites him most is bootstrapping: starting from a small kernel and letting an agent extend itself, such as building and deploying an integration Notion does not yet support. He calls coding agents the kernel of AGI, since code is a very useful primitive for representing deterministic logic. For knowledge-work agents, writing code is simply one more tool.

Asked how Notion fits among the labs and giant platforms, Last says the company sees itself as the Switzerland for models. Customers do not want to be locked into one lab's model, and which model is best changes month to month. Notion wants users to reach all the best models and switch easily.

Open-source models are part of that. He says there are four Chinese models that are quite good and much cheaper than frontier models; Notion had just added one to its agent the week before and plans to add all four. The role he describes is taking the best models, building high-quality agent implementations, and offering a collaborative workspace where humans and agents coordinate.

Notion's structure of blocks and databases is still very useful, but the agent is a new kind of customer. The default API used a verbose JSON format for blocks that was terrible for models, so the team created a Markdown dialect enhanced with Notion blocks and used SQLite for databases. Both work well because models are naturally good at them.

To find what works for agents, Last says the team is empirical: try things, notice when something uses too many tokens, and think about what the models were trained on. He jokes that it is user research where the user is always available, can be scripted and scaled, and just needs to be chatted with.

Last uses Claude Code or Codex in the command line and aims to have as many agents running as possible, all the time. Before bed he makes sure he has given them enough work that they will not be done by morning. His personal record is a coding agent running for about 13 days straight, and the host admits to waking at night to check on hers.

He uses Notion's personal agent daily, including to ask about how the custom agents launch is going. His favorite custom agent triages his work and personal email: it has a memory page, interviews him about what to archive, and builds a list of rules. After a couple of weeks he dropped the approval step, and it now archives everything he does not need to see, which he says solved his email problem.

A second agent triages internal product feedback and bugs posted in a Slack channel, routing each to the right team and filing a task in its database. It learns the same way and has built up hundreds of rules. His general pattern is to build a prototype, watch it in approval mode, and trust it after it has run many times, fixing it when it occasionally breaks.

To help non-technical teams, Notion runs workshops and hackathons. The people team has been one of the highest adopters of custom agents. Last thinks the interface is very human-like once people get past the basic concepts, and the main barrier is getting people to try and trusting that it will work.

Asked how his idea of Notion as a tool for thought has changed, Last says the goal used to be the best tool for humans to do their work directly, and is now the best tool for humans to manage agents doing it. He calls that a pretty fundamental shift. Yet most existing primitives still matter, and the team just needed a few new ones, such as a way to represent an agent.

Documents still matter because agents love to write Markdown, and databases still matter for structured data. If you run a swarm of 100 background coding agents, he says, you do not want 100 chat threads but a Kanban board. The coordination structure stays the same as before.

On a personal level, Last says he has not written code since the previous summer and does not type code anymore. The sequence went from typing everything, to tab completion, to talking to an agent in an outer loop, to now designing an end-to-end task with its verification and checking it at the end. He describes himself as an agent manager instead of a coder.