Pinecone's Vector Database Bet: Scaling Through the ChatGPT Tsunami

The episode opens by introducing Edo Liberty, CEO and founder of Pinecone, and framing the company as a vector database provider that has raised over $130 million, most recently at a $750 million valuation, and has become a core tool for AI app builders. The hosts note that Pinecone was already working on the problem before the generative AI craze and ChatGPT. Edo begins by saying vector databases are still vastly underutilized for semantic search, candidate generation, ranking, feed ranking, recommendation, and many other use cases.

Edo explains that when he started Pinecone, vector databases were a 'well-known secret' inside large companies such as Google and Amazon, which used them internally for search, recommendation, feed ranking, and anomaly detection. Educating the market was difficult, and investors would look confused when he said 'vector database.' He also had to overcome the fact that he had just come from AWS, where he helped build SageMaker, so many people assumed he was building an ML Ops product.

Edo describes the investor reaction with dry humor: after vector databases became a recognized category, some investors wrote to say they had always been fans, and others looked back at notes from three years earlier and realized he had said he would build exactly this. He jokes that it is uncommon for someone to say they will build something, actually build it, and succeed. He also notes that many had initially thought he was building an ML Ops platform because of his SageMaker background.

Once OpenAI pushed the topic into a wider audience, Edo says the technology did not change deeply for practitioners. What changed was the amount of oomph, capital, and energy behind it. Pinecone started being accessed by non-AI engineers, and that shift set off the scaling crisis that followed.

When non-AI engineers began using Pinecone, demand ramped so quickly that the company exhausted whole kinds of machines in GCP and AWS. Edo says Pinecone was spending millions of dollars a month on its free tier, and the entire team was dedicated to spinning up more environments to keep up. At peak, the company was seeing 10,000 sign-ups every day.

The load forced Pinecone back to the drawing board. Edo says the company redesigned the entire solution, later launching it as serverless, and the new architecture was maybe two orders of magnitude more efficient. It was not a strategic luxury; it came from necessity because the team could not handle the load.

Edo points to AutoGPT as a moment when usage completely blew up. He describes it as a minimal open source gadget, just a handful of Python files with 'spit and gum,' but it was a precursor to what people now call an agent. Suddenly the people onboarding Pinecone were not systems engineers building AI platforms but, in his phrase, dentists who remembered Python from college and wanted to play with AI.

Edo says millions of vectors is actually small scale today. When customers have millions of vectors, Pinecone tells them to knock themselves out, because it is hard to spend more than $100 a month. Pinecone truly shines at hundreds of millions or billions of vectors.

The target customers are often SaaS companies that build AI Q&A, semantic search, or deep AI solutions for their own customers' data. Notion and Gong are examples: each end customer might have a million documents, but the SaaS provider has 10,000 customers, so it may need to handle 10 billion embeddings across thousands of partitions cost-effectively. That is where multi-tenant economics become central.

Edo says the serverless architecture lets Pinecone run such multi-tenant workloads so that one paying end customer might cost a dollar a year, sometimes 50 cents. If a customer has a million vectors 10,000 times, the provider does not want each million-vector workload to cost $100; it wants each to cost 50 cents. That is the scale at which Pinecone's unit economics make sense.

On the application side, Edo says Q&A, semantic search, chatbots, and support bots are large and well-understood categories. Customers also use Pinecone for legal discovery, medical history discovery, image and audio and video work, anomaly detection, security, and some pharma and drug discovery applications. The meat and potatoes, however, remain text and images with search and RAG over them, and Edo argues RAG's magic happens at the semantic search layer rather than the reformulation layer.

Asked whether multimodal use cases are taking off, Edo says he has a mix of enthusiasm for tomorrow's promise and a healthy focus on yesterday's problems. Even with foundation models, the average company still finds it hard to train any deep learning model at all, even a million-parameter model, let alone a billion-parameter one. He distinguishes what research labs and technology leaders can do from what companies actually use, and says those can be five years apart.

Edo sees amazing multimodal results already, but he does not expect multimodal to hit the mainstream technology developer in the next year or two. That makes the near-term enterprise picture more text-first than the research agenda might suggest. The barriers are not only model capability but deployment reality.

One of the biggest barriers is hallucinations. Edo says large language models are designed to generate language, and when they are compelled to write a page about something they know nothing about, they will write a page, including made-up content. He compares it to the sixth-grade experience of writing an assignment while in a cold sweat, and says society and the technology world have to grapple with that failure mode.

Edo says the industry is only now starting to measure hallucinations. A model that always says 'I don't know' never hallucinates, but it is useless, so measuring usefulness, correctness, and faithfulness to the data matters more than a simple hallucination rate. He also points to a large effort around RAG, vector databases, and governed knowledge layers so data can be made available to models securely and with controls such as deletion for GDPR. He adds that Pinecone has run experiments loading a good chunk of the internet into an index and running a language model next to it with unsophisticated RAG, and that it can already outperform GPT-4.

The hosts ask about the explosion of vector database startups and incumbents adding vector support. Edo says storing vectors is indeed becoming a standard data type next to sentences and images: an embedding model produces a float array, and infrastructure needs somewhere to put it. But he argues there is real work required to query and work with that data type natively.

Edo explains why vector databases are unique: the numerical array becomes the key, the primary lookup, and the way data is organized on block storage, loaded into segments, and searched. He notes that this is how the brain does it, too, and warns that bolting the data type onto a legacy database and pretending it is a vector database will produce poor performance and serious issues. That is why he sees vector support as more than an add-on.

Asked what stack he would use to build a RAG product today, Edo says many people default to OpenAI, but he would not. He sees good performance from smaller, cheaper, sometimes open source models and says there is no reason to spend a ton of money. He likes Anyscale for the ML ETL and bulk data transformation layer, and mentions partners such as Percell, Cohere, AI21, and Hugging Face across embeddings, ranking, and summarization.

Edo says he does not yet have a good vendor or technology for evaluation and would like to see a leader emerge in that part of the stack. On data pipelines, he says companies such as Unstructured are becoming viable, especially for standard data channels. If data is imported with Fivetran from Snowflake into a common table, there are companies that will transform it and fit it into a vector database so teams can run RAG over sources such as Salesforce notes.

Edo gives an economics answer to where intelligence will live: the market must find a stable trade-off between cost, compute, infrastructure, and output quality. Running a 100 billion parameter model for every API call would bankrupt a platform and heat the planet, so that cannot be the long-term answer. With smaller models, he says, results can sometimes be better and are often not noticeably worse, at cents on the dollar in operating cost.

The hosts ask whether longer context windows make RAG obsolete. Edo says model companies allowing bigger and bigger context is unsurprising because their pricing model lets them charge more, but stuffing context has multiple problems: it is slow and expensive, it often does not help, and it eventually runs out of steam. Ten job descriptions might fit in a prompt, but a million do not, and sending all company data to OpenAI on every query is not the right approach.

Even when a context window is theoretically workable, Edo argues retrieval can send 10,000 tokens instead of 100,000, or 3,000 instead of more, with equal or better quality and much lower cost. That makes retrieval an economics decision even at small scale. Most companies, however, are still trying to get the basics to work rather than iterating deeply on embedding models, reranking, and filtering.

Edo tells larger companies that RAG is not magic: it takes scientists and engineers working for multiple quarters, learning and iterating, adding and removing components. Every use case differs in speed, cost, accuracy, format, length, and data freshness. The most common failure he sees is cost miscalculation, where a company does napkin math, assumes Pinecone will cost $50,000 a month, and abandons a project that would actually cost around $500 a month.

The hosts ask whether shipping a more efficient serverless product could reduce Pinecone's revenue. Edo says the transition was very painful: revenue that had been going up and to the right suddenly flattened, even as workloads grew faster, because the product became much cheaper. His advice to founders is to make that transition earlier rather than later and take the hit as soon as possible.

Edo says Pinecone thinks it is leaving more than half its revenue on the table. For some customers the cost reduction is 70, 80, or 90 percent, and for storage-heavy pharma workloads the new spend can be as little as 1 or 2 percent of the old spend. He cites companies that used to pay $100,000 a month—he corrects himself to a year—and now pay almost nothing; investors are not happy, but he argues it is the right thing for customers and for people building on the platform.

Edo frames the move as necessary because the AI stack is still in its first innings. He expects almost every company eventually to have a vector database doing something small, medium, or large. If Pinecone needs to fit snugly into the cost structure of tens or hundreds of thousands of workloads, it has to accept medium-term pain.

Asked where new startups should build, Edo says he is much more excited about the application and solution space than the infrastructure layer. Infrastructure has a winner-take-all dynamic that is good for Pinecone but limits room for new companies, while applications have enormous energy and many problems that can be solved in creative new ways. He describes a conveyor belt of innovation from tiny startups to enterprises, with hundreds of companies a week doing unique and interesting things.

If Edo were not building Pinecone and were building an AI application, he says he loves communication data: email, Slack, Twitter, meetings, transcriptions, and conversations. He finds human communication, written or spoken, fascinating because there is a ton of knowledge to extract and it is messy, exciting, complicated, and rich. He says it would be something like that, whether the data is medical records, emails, or Jira tickets.

Reflecting on his time at AWS, Edo says large companies have different constraints: at AWS scale, a product that does not bring in hundreds of millions of dollars a year is a failed product. Startups have a different horizon and appetite for risk, and he says he does not know whether Amazon, Google, or Microsoft are even working on what Pinecone is working on. Pinecone is focused on things it knows will be part of the stack and may hit the market in a year and a half or two years.

Looking ahead, Edo expects a significant shift in hardware because the current GPU situation is not sustainable; he imagines CPUs, GPUs, and specialized servers optimized for training or serving. He also expects data pipelines and data management to change because tools from five or ten years ago no longer cut it in operational headache, cost, or waiting time. Finally, he expects more governance and visibility systems so companies can control a stack that today runs open loop for most companies.

In the quick-fire round, Edo calls foundation models overhyped and code generation or coding assistance underhyped and exceedingly useful. The biggest surprise in building Pinecone was a complete rewrite of the database in Rust after about a year and a half or two years; he thought it was suicidal, but CTO Ram Sriharsha said it would be done in a month, and it took two or three months with a much better result. He also says agents already work to a degree, and that the most common failure mode is doing nothing. The hosts close by highlighting his view that applications may be more attractive than infrastructure and his skepticism about the sustainability of bigger models and more GPUs.