Inside Notion AI: Prototypes That Annoy Everyone, Evals Built In-House, and Models Rented from Partners

Linus Lee joined Notion's AI team in January 2023, a few months after the company began working on the technology. The story he tells is that around the time GPT-3 dropped, Notion's co-founders skipped part of a company offsite to hack on it in a hotel room. That weekend produced the first version of what the team internally called Notion AI Writer.

Writer went to beta in November and shipped in February. It lets you summarize a page, pull out action items, fix grammar, or draft marketing copy inside a document. One lesson from the beta was that people and AI iterate together, so the team added conversational follow-ups such as "make it shorter" or "make it punchier."

The second product, Autofill, brought AI to Notion databases by filling whole columns, such as key topics for meeting notes or core needs extracted from user-interview transcripts. It arrived in May alongside Notion's project-management suite. The third, Q&A, launched the week of the recording and answers questions across an entire workspace.

Q&A came from two converging paths. Customers kept saying that as workspaces grow, information gets hard to find, especially when a colleague wrote something you didn't know existed. At the same time, retrieval-augmented generation was emerging in the wider AI community as a way to ground models in your own data.

Lee describes Notion's process as repeated cycles of expanding and contracting. The team starts with broad problem statements, such as helping people stay organized or find information. A few people then quickly prototype different techniques, and because Notion runs on Notion, the prototypes get used internally right away.

By the time several prototypes exist, it is usually clear which approaches are promising. The team then recalibrates with better knowledge of both the user problem and the strength of each approach. Only after that do they commit to a direction and iterate until they trust both the interface and the output quality.

The early Q&A prototype ran on every question anyone asked in the company, and it was irritating enough that colleagues asked the AI team to switch it off. Lee says that pain was the point. Any improvement or regression was visible every day, and real usage in people's normal work made it obvious whether the thing was useful.

The exploratory phase needs people who are enthusiastic about the technology, who know what techniques are being tried, and who will push through technical and operational obstacles. Once the solution is better defined, the work looks like ordinary product building, with user research and designers involved.

The AI team has about a dozen people, split fairly evenly between those who work on what Lee calls quality, meaning how correct and coherent model outputs are, and those who work on product concerns such as interface and integration into Notion. A couple of designers collaborate closely, checking in often during ideation and iterating every couple of weeks.

Ownership is feature by feature. The AI team owns the Writer bar and the Q&A chat box, while Autofill is owned by the databases team, with tight collaboration on hard AI problems. Lee says Notion has not settled whether a central hub with liaisons or AI engineers embedded in every team is the right long-term shape.

His thinking on retrieval has changed. He first saw it as a purpose-built tool for answering questions, but now believes it augments every capability, including features that surface related information or recent changes without ever producing natural language for the user. That makes it a foundational piece, even if a Q&A-specific team existed.

Lee compares the situation to the DevOps era, when everyone was told to do ops yet infrastructure teams still mattered. He bets language models will end up similarly: many engineers will use them, but a central group will still handle monitoring, quality, datasets and training tooling.

Internal dogfooding is the richest source of ideas, but Lee cautions that Notion uses every feature in an unusual way and that many customers, including students, work differently. So the team also relies on early testers and Notion ambassadors, who often push features into use cases the team had not anticipated.

Translation is the example. After Autofill launched, people abroad used the custom prompt field to translate pages and summaries, so translation became a built-in prompt. Lee says this mirrors a broader Notion pattern: expose a fully customizable layer, watch what people do with it, then bake the winners into templates.

Usage follows a power law. Summarization, improving writing and fixing grammar are the most popular built-in uses, and translation is big in some locales. Yet a large share of tokens goes to a prompt that revises earlier output, which shows people use the presets for inspiration and then iterate toward their own needs.

Power users often end up with handwritten, reusable prompts. Lee mentions a newsletter writer with a template that turns each issue into a social-media image tag. The host links this to the blank-canvas problem, and Lee adds that suggesting likely next revisions would help, because coming up with something to say to the AI is real mental work.

Lee says evaluation was as hard as expected. Writer has plenty of wiggle room, because many different essays satisfy a request. Q&A asks for one specific answer, and the model can fail by saying the wrong thing or by not looking at the right document.

The boundary of what the model should answer was also fuzzy. Users asked meta questions such as how to share a page with a colleague, or how many people are in the workspace. Another tough category was time-based questions like what the marketing team is working on this week, since no single document says that and the system must reason over updates across time.

The remedy is to enumerate the kinds of questions that stress the pipeline, build high-quality evaluation sets for each, decide on grading criteria and then fix edge cases. Lee says this decomposes into around 30 sub-problems. On top of that, nobody in the industry had clear answers about customer needs for privacy, security and scale, so Notion went back to customers and started from first principles.

Most of Notion's tooling was built in-house. Part of the reason is timing, since few tools existed when they started. Notion pages are also complex structured documents with tables, columns and metadata like tasks and due dates, which off-the-shelf tools did not handle well. Owning the tools also makes small changes cheap, such as adding a third column to compare models side by side.

Lee frames evaluation as a spectrum from cheap deterministic checks, through model-graded and human-annotator review, to ML engineers reading raw outputs. The last is the most expensive but pays off, because it reveals why a model fails, for example with relative dates versus absolute ones. The team can then intervene at the right stage: embeddings and ranking if the right document never reached the context, or the answering step if it did.

Notion has worked with both Anthropic and OpenAI since early on. Lee argues that competing with them, or Google, on infrastructure and base models would be exceedingly difficult. In his view their problem is building a good instruction-following model and hosting it reliably, while Notion's problem is understanding its particular tasks and collecting or generating the data for them.

Notion has committed to not training on customer data, which Lee supports personally because he keeps much of his own information there. The constraint pushed the team toward synthetic data, which in turn forced a systematic understanding of the archetypes of Notion pages. Nothing based on open-source models has shipped to production yet, though the team is exploring some.

Defining a task well matters. A good summary of a meeting note differs from a good summary of a technical document, and a bug-report page may not need summarizing at all. Model choice then depends on capability, throughput and cost: Autofill re-extracts information whenever a page changes, so it wants high throughput, and different features may use different scales or providers.

Prompts are also wrapped in a lot of Notion's own processing. Writer wraps your revision request in a template with the dialogue history, and Q&A supports multi-turn conversation with a query-rewrite step, so a follow-up about "that automation project" becomes a proper search. On hallucination, Lee says the team has made strides but is still working on the edges.

On switching providers, Lee says prompts are downstream of evaluation criteria, which come from understanding the task. That understanding carries across models, while a few per-model quirks need tweaking, such as one vendor's model that keeps writing "in this document." Large models also transfer well across languages; Notion prototypes in English and builds multilingual evals closer to launch, and Q&A can even read Japanese documents and answer in English.

On balancing freedom and guidance, Lee says the answer for AI mirrors Notion generally: choose the right building blocks. Power users can drop down to a nearly blank prompt, though Notion still adds page and workspace context. Notion's rich block abstraction is why users never write HTML, and Lee says the team has no consensus yet on what the equivalent blocks for AI are.

The right abstraction depends on the output. Free-form writing is well served by custom prompts. For generative interfaces, where a model outputs UI, he sees a need for a component library so buttons look coherent, and he notes Adept's domain-specific language for taking actions in the browser as another example.

In the quickfire round, Lee names context length as the overhyped idea. He struggles to picture useful tasks needing 50,000 words, noise in a large context still hurts, and retrieval is always where you start. He guesses 16-32k tokens may be enough for most tasks, and he notes that long-context training data is hard to find, with synthetic needle-in-a-haystack tricks as one workaround.

His underhyped pick is alternatives to Transformers. Transformers are excellent at modeling long sequences efficiently, he says, but nothing about them is architecturally optimal, and state-space models are the ones that look cool to him. His biggest surprise is how well general approaches work, so he bets on training one model across a product's tasks and giving the AI more control over generated interfaces.

Asked where else he would work on AI features, Lee picks Midjourney for its lab-like way of working, where individual champions push their own prototypes, similar to what GitHub Next did. He argues individuals iterate faster than teams. He also admires Adept for tackling an ambitious general agent that can do any computer task.

In the debrief, the hosts highlight how much Notion dogfoods both features and evaluation, and how the team seems to be moving from per-product AI toward a horizontal platform with shared retrieval and perhaps a single model for many tasks. They also note how hard it is to predict usage: Q&A, a search-like product, is being used for questions about notion itself, such as how to share a page.

They find it notable that Notion built so much itself, especially evals, in a market that has since produced many startups. They read Lee as saying that owning the stack let them iterate faster, and that reading the failing cases matters more than summarizing quality as a number. They also think it makes sense that an app company outsources models to Anthropic and OpenAI rather than training and deploying at scale.

The conversation then turns to OpenAI's Dev Day. The hosts see the main trends as longer context windows and cheaper models, with OpenAI moving further up the stack. They note that a price cut can swing a business's AI costs from a large share of revenue to a small one, and that this makes it sensible to build features that look too expensive today.

On the claim that Dev Day would kill vector databases and startups, the hosts are skeptical. They argue companies will still want control over where data goes and how retrieval is customized, and they liken OpenAI to AWS, which offers many tools while leaving room for startups to win in specific verticals, as Snowflake did.