The Search Engine Under AI Apps: Why Bigger Context Windows Aren't Enough

Jacob Effron introduces Simon, Turbopuffer's CEO and a decade-long Shopify infrastructure veteran, and notes that Turbopuffer powers AI applications such as Cursor, Notion, and Linear. The conversation opens with the problem of connecting business-specific context to AI applications, whether by stuffing context windows or adopting a new search database. Simon frames the discussion around the origin of Turbopuffer and the search paradigm it is built for.

Simon explains that the first vector-database wave was driven by small context windows of roughly 8K tokens, since even many articles are larger than that. When context windows grew, many early applications could stuff them and run for product-market fit, which he says is exactly what he would have done too. But companies now want to connect all of their data to LLMs: tens of millions of tokens, permissions on documents, high recall, and economics that work per user.

Even with a million or 10 million context tokens, Simon says companies want to connect all their data to LLMs, often more than a million tokens and sometimes tens of millions. They need high recall and permissions, and they need per-user economics that justify storing the data. On traditional architectures storing every byte on three disks can cost north of a dollar per logical gigabyte, while object storage can make searching hundreds of millions or billions of tokens economically viable.

He uses the acronym SCRAP to describe the limits of large context windows: scale, cost, recall, ACLs, and performance. Attention is still quadratic, VRAM is one of the hottest commodities on earth, and needle-in-a-haystack benchmarks do not prove reasoning over very large corpora. Permissions still need enforcement rather than a model's promise, and sub-second responses are hard when a frontier model must load a huge context window.

Simon traces the why-now to a new storage architecture that has already appeared in streaming, observability, and OLAP companies. NVMe SSDs became fast and cheap enough: cloud instances offer them at roughly 5 to 8 cents per gigabyte, and they can drive about 10 GB/s versus roughly 100 GB/s for DRAM. A gigabyte of RAM in the cloud costs around $5, so the bandwidth gap is about 10x while the cost gap is about two orders of magnitude.

That trade-off only became usable when storage engines were written to take advantage of it; Simon notes that io_uring is recent and that the Linux page cache is too slow to keep up with these drives. NVMe became generally available on AWS and GCP in the late 2010s, with roughly mid-2017 on AWS and mid-2016 on GCP for one SKU. S3 then became read-after-write consistent in December 2020, a core primitive for building a database directly on object storage.

The final primitive was compare-and-swap, which S3 released only about six months before the summer-2025 recording; GCP already had it, which is why Turbopuffer started there. Simon says these three pieces—NVMe, S3 consistency, and compare-and-swap—make it possible to build a database with everything on object storage. The main trade-off is write latency: committing to S3 has a P90 of roughly 100 to 200 milliseconds, which is too slow for a Shopify-scale checkout but usually fine for search.

Simon says Turbopuffer is mainly a search engine, and that applications modeling data and permissions should still use a relational database. A million vectors can reasonably live in a Postgres extension, but tens or hundreds of millions, billions, or hundreds of billions change the calculation. Vectors are large—a kilobyte of text can turn into 30 kilobytes of vector data—and the economics of replicating them across three disks at roughly 60 cents per gigabyte all-in are hard to escape.

Cursor was Turbopuffer's first paying customer and uses it for semantic search over one or many codebases, with vectors obfuscated and encrypted. Notion uses Turbopuffer for Q&A over its wiki and canvases, including questions about leave policy or family leave options. Linear uses it for search and for similarities such as duplicate issues or finding the person who should work on a type of issue.

Asked about incumbents such as Elastic that combine vector and traditional search, Simon says traditional architectures have lower latency and more features but may not fit the cost profile of searching trillions of documents. If per-user economics allow keeping everything in memory or on disks, that can make sense. Turbopuffer tries to price against the first-principles cost of its infrastructure.

Simon says simplicity continues to surprise him: keeping everything, including metadata, on object storage was partly forced by fast-growing customers, but it produced a simpler system. Turbopuffer has millions of S3 prefixes and sometimes lists them for background work; because S3 listing is slow, it writes read-through cache files every few pages so it can list from many keys in parallel. He adds that 404s are free, compare-and-swap exists, and other S3 headers let the team build a low-latency, durable system.

On vector search, Simon says the only exact way to return the 10 nearest vectors is to inspect every vector, so systems approximate and measure recall as the overlap with exhaustive search. Turbopuffer finds customers feel good at around 95% recall, the point at which retrieval is less likely to be the reason evals fail. It runs exhaustive searches on a percentage of production queries in the background to report actual recall.

The hardest problem is incrementally maintaining that index with high recall as data changes—for example, when an e-commerce store selling pants and dresses starts selling shoes. Early Turbopuffer rebuilt the entire index after a percentage of data was overwritten, as many production systems do, but that was expensive to scale. Simon says Turbopuffer now incrementally maintains the ANN index into hundreds of millions and billions of vectors, while keeping shards as large as possible because sharding early is costly.

Filtering is the second hard problem: a vector query often also asks whether an item ships to Canada or is red, which requires different index structures. Rare filters can be handled by pre-filtering, and common filters such as public products can be handled by overfetching and removing non-matches. But a filter that applies to roughly 50% of data and is randomized across clusters can wipe out the nearest cluster, so the query planner must understand both filters and vectors to preserve recall.

Simon says simplicity and focus are core cultural values, and Turbopuffer asked what its customers' hardest problem actually was: not choosing or running an embedding model or reranker, but storing petabytes of data and searching it. Customers now ask for help ingesting 100 million vectors quickly, but running embeddings would pull Turbopuffer into low-latency GPU operations and the GPU procurement game. The company would rather do a really good job on the things it puts its name behind.

Reliability is number one and simplicity is number two because Simon spent almost a decade on Shopify's last-resort pager and has debugged databases alone at 3:00 a.m. He says Turbopuffer's only stateful dependency is object storage, so all nodes can be blown away and autoscaled down without losing data or waking customers. That architecture is meant to keep Turbopuffer from waking its customers up at night.

Turbopuffer advises customers on embeddings and rerankers but does not want to own them; Simon notes that some embedding models take 300 milliseconds per query, while Turbopuffer takes about 10 milliseconds, so slow embeddings can paint a search product into a corner. He also prefers customers writing a search.py or search.ts over maintaining a massive search DSL with thousands of lines of JSON. Writing code gives teams control, tests, and evals for only a small performance penalty.

Some features do need engine support for performance, such as late interaction where a query searches with 128 queries at once and is too heavy to funnel over JSON. Simon frames software design through European cities built incrementally, and says Ottawa's downtown is less beautiful because too much had to happen quickly. Guessing too much about future usage leads to bad software, so Turbopuffer tries to avoid overbuilding.

Jacob notes that enterprises often want one end-to-end RAG system, and infrastructure companies get dragged into adjacent parts of the stack because that is what buyers want. Simon agrees that everyone bundles eventually as common customer patterns emerge, but says a company can get too greedy and do too much too soon at the cost of the focus that created the initial magic. Turbopuffer expects to partner and work on pieces that make the end-to-end process easier, while being disciplined about saying no.

Simon describes a grab bag of ideas and a process of Bayesian gradient descent on what matters this month, continuously updating priors based on customer demand. The difficult part is deciding when to pull something out of the grab bag and not saying no forever. In practice, Turbopuffer often says yes to more performance and lower cost on the core product instead of expanding scope.

Asked which parts of the AI infrastructure stack will persist, Simon says he likes companies that have state, which is why Turbopuffer is stateful. Workflow companies eventually capture state, and stateful companies eventually capture workflow, but trying to do both at once requires years of lab R&D before going to market. He sees frontier labs choosing best-in-class Lego pieces, specialized AI companies doubling down on niches, and bundling arriving later than it did for the previous generation of software companies.

Simon says memory implementations start with the simplest possible thing: long-running coding agents compact context into a text file or ask the LLM to summarize. Then applications add lateral memories across chats, such as a shared ChatGPT account that remembers gardening advice and assumes the user is a Rust programmer. Those memories are not a lot of data, so pulling them all into context or doing simple similarity can be enough.

At larger scale, Turbopuffer sees customers with long-standing conversations with users, which raises a sliding-window question between what counts as a memory and what is just prior context to search. Simon says Turbopuffer has not seen enough customer patterns to justify shipping a specific memory product. The same data could sit in Turbopuffer as a KV store on object storage, with or without vector search or keyword search.

He expects implementations that search over the entire history to outperform those that condense everything into memories, although condensed memories carry higher weight because they are compressed from a chat. At the same time, he would not pretend to know exactly where memory is going. The area is still full of experiments, and the data volume ranges from small to very large.

Simon argues that a good database company needs a new workload and a new storage architecture, and Turbopuffer's hypothesis is that connecting large amounts of data to LLMs is that workload. Even very capable models will need a pipe into something computationally good on the other side, because much of the data is not needed at all times and fuzzy search is often required. He thinks that architecture could look a lot like Turbopuffer, while many customers also use it for traditional search.

He compares AI features to mobile apps: once mobile became table stakes, serious software companies stopped debating whether they needed a mobile app. Simon expects semantic search, similarity or dedupe or recommendations, report generation, and agentic workflows to become table stakes AI features in SaaS. Incumbent SaaS providers are building these features well, and upstarts have an opportunity to build interfaces native to them.

On multimodal data, Simon says Turbopuffer can handle it and object-storage economics make it attractive to embed both product images and descriptions, but he sees fewer companies doing multimodal over images and attachments than the market chatter suggests. The economics could let customers embed all PDFs without worrying about a 2,000-page PowerPoint upload, which is otherwise scary for SaaS companies that do not want usage-based pricing. In the quickfire round, he says the most interesting place to run AI would be a frontier lab such as OpenAI or Anthropic to see models three to six months out, and he admits the name Turbopuffer came from something that sounded funny and had a good emoji; the marketing story is that a deflated puffer fish is on object storage and an expanded one is in DRAM, with SSD in between.

Asked what he has changed his mind on, Simon says he keeps being surprised that the simple object-storage approach continues to work, and on mistakes he says Turbopuffer has not made a major product mistake yet; early customers used every feature, so every line of code was in production. As a founder, he has learned to trust his instincts and give the team permission to try things even when others, especially VCs, say to do embeddings or reranking; the vibes have to be right. If he could ask someone from the future one question, it would be how much the agent is searching and how much it uses a search engine rather than loading everything into context. He also says he would have loved an LLM when he was 11 learning to program from a small Danish web, and he finds joy in how much access a curious child such as his daughter now has.