Bolt Memory onto GPT-4 and Cut Hallucinations by up to 50%

Sarah Guo introduces Edo Liberty as founder and CEO of Pinecone, a vector database meant to give AI applications long-term memory. Before the company he directed research at AWS AI Labs and, earlier, at Yahoo. Vector databases, he explains, operate on data that models have already analyzed. A language model or any other model is a mathematical object. When it reads a document, a paragraph, or an image, it does not store the pixels or the words. It stores a numeric representation, an embedding, and that is the object Pinecone stores, retrieves, and searches at scale.

When they opened the category, people called him concerned: what is a vector, and why start a database? They do not ask that anymore. Gil wants the 2019 version of the story, three years before ChatGPT, when the generative wave had not arrived. Liberty's image is geological. The tsunami of AI had not hit yet, but in 2019 the earthquake had. Deep learning was already in hand. Transformer models like BERT were reaching more mainstream engineering teams. You could already connect the dots and see where this was going.

Knowing the direction did not quiet the usual founder anxiety. For several months before starting Pinecone he swung wildly between two fears: they were already too late, or nobody knew what this was and they were way too early. Then he noticed the pattern. Having both the too-early and too-late mood swings, he figured, maybe meant it was exactly the right time.

Guo asks for the jobs embeddings actually do, from reliability to context length. The obvious way to add knowledge to a conversational agent, Liberty says, is not shocking: bring the relevant information into context. That corpus might be a company's own data or the internet. It happens that LLMs are already very good at representing data the way they want to consume it, as embeddings, so at question time you can retrieve what is associated, correlated, or similar and hand it to the model. The result is more accurate answers. People now brand this generative AI; he notes the retrieve-then-generate pattern is more general than chatbots.

As a side experiment, Pinecone loaded Common Crawl, the frequently recrawled top internet pages, into the database and augmented GPT-3.5, GPT-4, Llama, Mistral, and models from Cohere and Anthropic. Even on internet data those models had already trained on, RAG reduced hallucinations significantly, up to 50% in some cases. Many of the models also started behaving quite similarly in accuracy, even though without RAG their behavior had been quite different. It was a uniform improvement and a leveling of the playing field. Do the same thing with proprietary company data the models never saw, and quality, he says, ends up incredibly high.

Most customers are not retrieving against the public web. They bring legal files, medical records, internal wikis, sales calls, an infinite variety of company data. Liberty is careful not to oversell the label: this is semantic search, and RAG is only the application they are focusing on in this conversation. Text is still the dominant data type even as image models get good. The pattern he sees over and over is companies that hold a lot of their own users' text and want to search or generate on top of it. Notion's Q&A runs on Pinecone for tens of thousands, probably hundreds of thousands, of Notion's customers. Gong does the same thing with sales calls.

Guo folds two February announcements together and calls the serverless offering Canopy. Liberty separates them on the spot. Canopy is an open-source course, a framework so people can learn RAG. Pinecone serverless is just Pinecone, running serverless. What it is for is removing the limits people used to hit. Early applications were recommendation engines and anomaly detection: fairly small scale, super-low latency, sometimes high throughput. Other databases still play that field. Pinecone's own customers showed that something else mattered more: scale and cost.

If you want to answer correctly, you have to know a lot, which means ingesting hundreds of millions, billions, sometimes tens of billions of vectors, then querying them without exploding spend, and doing it without weeks or months of setup. The old architecture, and frankly any architecture that is not serverless, makes that hard. Serverless is easy to operate and, he says, has no theoretical scale limit. They have tested it with tens of billions of vectors on live customer traffic. He will not walk through the design, only that it is meant to be asymptotically more efficient than other architectures, so foundation models can have all the information they need sitting ready.

Later he adds that serverless is the fourth almost complete rewrite of the entire database. Canopy exists because Pinecone sits deep in the stack. Building something like Notion Q&A on top still means ingesting documents, chunking them into factoids, embedding, writing to the database, rewriting and embedding the query, searching, and reranking. People told the company that system is quite complex, and they are right. Canopy is an end-to-end cookbook: take this, it should work. Then you have to make it better for your application, because medical data is not Jira tickets, and Jira tickets are not Slack messages.

The two comparison points developers reach for are traditional databases, Postgres with pgvector, and incumbent search, Elastic or Algolia. Liberty goes back to the job: give foundation models as much knowledge as possible, easily, at scale, on a budget, with unit economics that work for an AI product. Those other products do not work for that job. Vector indexes on classic databases are a bolt-on, retrofitted feature. They are fine at small scale. In production the efficiency, scale, and cost tradeoffs show, because the systems were designed to do something else.

Keyword search is the wrong mode. The contextual space where text, documents, and images live is a high-dimensional numeric space, not a keyword space. Anyone who has searched their inbox for an email they know exists and failed to find it already knows how deeply flawed keyword retrieval can be. Tiny workloads and first tutorials on embeddings might still be fine on the database you already have, including MongoDB. People hit the wall when they try to launch a product with real unit economics. Plenty start on Pinecone anyway, as enthusiasts who like learning a new kind of database.

Pinecone still supports hybrid search, because getting embeddings to perform perfectly can be intricate, and scoring keywords alongside them is convenient. Under the hood, any keyword search is sparse vectors, so they generalized the interface: send sparse vectors, dense vectors, or both. Liberty would not bet on baking keywords into search as the dominant mode. Hybrid will remain as boosting and other levers to control results. The mode of stuffing keywords into the query, he thinks, is going away.

Closed source is the other database-market question. Most mature databases opened source before the cloud was a mature platform; that was the only way to put a technically complex product in engineers' hands, a precursor to product-led growth. Pinecone asked why people open-source: trust, community contribution, and a channel to users. Trust, they decided, can come from being excellent at the service. Contributions, even at open-source database companies, come in the high 90s of percent from the company itself. A fully managed multi-tenant system this complicated would be a Herculean effort for anyone else to run. Guo recalls Ali Ghodsi of Databricks comparing successful open source to a hole-in-one, after which you pick up a baseball bat and have to hit a grand slam on the business model. Liberty agrees 100%. Newer vector-database players all took the open-source angle against Pinecone, and even the young ones are already struggling with that strategy.

Long, even supposedly infinite, context windows are the feature that might look like a bypass of vector databases. Liberty does not know what infinite context means, except that it is very big and never ends. He offers three reasons not to panic. First, model vendors charge by the token, so letting you use a huge window is good for their business. Second, there is plenty of evidence that stretching context does not improve results unless you do it very carefully; context stuffing just makes you pay more. Third, even if you buy the marketing, the logic runs its course. It is like saying you do not need Google because every time you query you can send the internet along with the question. Theoretically possible, practically not.

At some point the window is gigabytes, then terabytes, and you have to stop. Already today some users run models of only a few billion parameters next to a vector database that contains trillions of parameters, and they get better performance that way. Attaching all context to everything you do, he thinks, runs its course very quickly, and it is also unnecessary.

Gil turns to leakage. If you fine-tune an HR chatbot on company data, salaries should not appear in someone else's answer. Leakage, Liberty says, happens two ways: a foundation-model vendor that retrains or records your data opaquely, or an in-house app whose fine-tune pops sensitive facts into the wrong questions. The vector-database pattern is almost simple-minded. You do not fine-tune the model on proprietary data, so you know it contains none of it. At retrieval time you fetch the right rows, run inference only, and do not save the interaction. The data is ephemeral. A legal team that deletes a document deletes it from the vector database, and the model never has it again. No forgetting mechanism required. That operational sanity, plus GDPR-style deletion, is one of the main reasons people attach a vector database to a foundation model.

The three overlapping tactics are prompt engineering, fine-tuning, and RAG. As a scientist, Liberty is all for fine-tuning; done right, the evidence says it helps tremendously. As a business owner, he says it is extremely hard to do well. Without a research team you might make the model significantly worse, and more data often regresses it. Prompt engineering is necessary if you need a format or a property, then it runs its course. RAG is incredibly powerful and still not simple. It is simpler than the other two, and it still takes work, understanding, and experimentation. When the simplest solution is still somewhat complex, that is almost the hallmark of a nascent market.

What is next for Pinecone is the unglamorous infrastructure list: ease of use, security, stability, cost, scale, performance. Serverless is becoming faster, bigger, better, more secure, easier. As an engineer he is starting to grapple with problems that very large, trailblazing companies hit, and with issues the information-retrieval community has worked for 40 or 50 years, now showing up again inside RAG and AI. Getting models to be truly knowledgeable is still complex.

Asked what excites him in AI aside from databases, he wants a distillation of foundation models, and he does not mean the usual compression. He means separating reasoning from knowledge. Foundation models, in his view, get that split fundamentally wrong. Today's tools are crude, and we try to use everything for everything. Shockingly, that kind of works, because the tools are efficient and general-purpose. They are still blunt instruments. As a technologist it hurts the brain: we take half the internet and cram it into GPU memory. That cannot be the right thing to do.

Once the community learns how the subsystems should interact, and how to build each one to do its role, the same tasks get cheaper, faster, and better, or the same resources achieve much more. Last year was a year of experimentation. This year, he thinks, people will take those experiments to production and build products, which should be exciting for consumers of AI. Guo agrees it is a very exciting year ahead.