GPT-4 Gets Half Your Data Questions Wrong. Snowflake's Answer Is Systems Design

The clip that opens Sequoia's Training Data conversation is a number, not a slogan. Snowflake CEO Sridhar Ramaswamy says a reliable "talk to your data" application is the product that even companies armed with GPT-4 and "an army of software engineers" still cannot quite ship. Used out of the box, GPT-4 lands at "45-odd percent" reliability on those questions—it gets roughly half of them wrong. Snowflake, he says, is already well into the 90s and racing toward 99%.

The catch, which he is happy to admit, is that Snowflake restricts the domain and treats the problem as software engineering rather than a pure model race. That restriction is exactly what makes customers sit up. Teams with money and headcount still hit a wall they do not expect to break through on their own. For a company sitting at the center of enterprise data, reliability is not a side quest. It is the product.

Hosts Sonya Huang and Pat Grady introduce Ramaswamy as someone they have known for two decades—he started at Google on the same April 2003 day as Sequoia partner Bill Coughran. He is a computer scientist and, in his own joke, a "reformed academic" who left research because he wanted more impact. He ran Google's search-ads business for close to a decade and then all of ads and commerce for five years. That franchise grew from $1.5 billion to more than $120 billion in revenue on his watch.

Sequoia later backed Neeva, his attempt to rethink search, before Snowflake acquired the company and he succeeded Frank Slootman as CEO. He describes Snowflake as "the AI data cloud." The core thesis is that a cloud computing platform with data at its center will serve enterprises better than a generic cloud, especially as AI changes how data is stored, moved, and accessed. The company has more than 10,000 customers and did $2.6 billion in revenue last year, sitting, as he puts it, at the center of everything enterprise and data.

Unlike mobile or the browser, Ramaswamy says almost nobody still needs convincing that AI will be transformational. Anyone who has asked ChatGPT to write a poem or make an image has felt the magic, so awareness inside enterprises is already "incredibly high." Snowflake has thousands of customers at various stages of implementation. Bayer wants to put data in business users' hands without waiting a week for an analyst and a BI tool. Unstructured data that once required a software project—an image, a transcript, a contract—can now be fed to a model and queried, and Document AI pulls structured fields from contracts and receipts so the "magic numbers" sitting in company folders can actually be analyzed.

He calls the demand pattern a sandwich. The bottom layer is transforming data more flexibly; the top layer is making that data interactively available to business users. Snowflake's first move was to infuse AI into the platform rather than bolt on another service. Cortex AI is a model garden, but more importantly it is built into Snowflake, so any analyst who can write SQL automatically gets AI. Early apps like Document AI sit in that progression: Iceberg makes storage interoperable, and two commands replace what used to be a software-engineering project. "You have no business believing the raw output of a language model," he says, because it is ungrounded and does not know truth from falsehood or authority.

Customers keep reminding him that 100 to 1,000 times more data sits in cheap cloud storage than inside a specialized system like Snowflake. Industry taste has shifted toward interoperable data that many applications can read. Snowflake pushed Iceberg as that format and announced a cloud catalog, betting that in ten years most data will live in cheap object storage, in open formats, reached through open catalogs. Some customers already run video models via Snowflake's container services on objects in S3, extract transcripts, and write them back. The surface area of data Snowflake should play in is expanding on purpose.

Grady lays out the practical spectrum: send data to OpenAI and worry about leakage; pull a model off Hugging Face and build everything yourself, which is safe but painful; or sit in the middle with Amazon Bedrock or Snowflake. Ramaswamy says security, governance, and ease of use come out of the box, along with collaboration and third-party apps. Recreating access-control lists, standing up a bespoke vector index, picking a model, and writing custom LangChain routing is a project he is sure 99.9% of customers "want no part of." They wanted a chatbot on 100,000 documents to replace an annoying FAQ box, with prior governance intact and a guarantee that customer data never trains a cross-customer model. Snowflake's magic, he says, is making the hard simple: what used to be a software project with data-security risk becomes about six hours of work for an analyst, and his ten-year bet is that ecosystems keep moving upstream the way nobody changes a carburetor anymore.

The application that makes even GPT-4-rich engineering armies flinch is reliable talk-to-your-data. Out of the box the model still sits around 45% reliability. Snowflake is in the 90s and chasing 99%, because it restricts the domain and turns the job into software engineering. Customers perk up because they recognize a wall they are unlikely to break through alone, no matter the budget.

How? "It's systems design." The same way an effective coding copilot works in practice: break the problem down, present the right context, and do not throw a giant model at everything. Deciding whether to answer a question is a different subtask from deciding how to answer it, so different models can specialize. Cortex Analyst is more restrictive than a free-flow chatbot. It sits on the customer's schema, historical queries, and semantic context. Schemas are weird, column names lie, and every company has its own definition of revenue. Take the best model on the planet and let it loose on an arbitrary schema, he says, and the chance it understands the nuance is "close to zero." Big deployments have 200,000 tables, and tens of thousands of those tables have the word "Revenue" in the name without sharing a meaning.

They also train the system to refuse questions instead of pretending it can answer everything. Any competent ML engineer will hand you an AUC curve: squeeze the model to do more and it gets less reliable. You can get 100% precision by answering nothing, which is useless. The trade-off between reliability and the ability to respond has to be a conscious product choice. Ramaswamy calls that a product-structure problem. Great founders and product managers are people who can see the right constraint so the thing is both useful and trustworthy.

Grady has noticed Snowflake's product velocity inflecting over the past six months, which is the opposite of the usual large-company story. Ramaswamy says he has run this play before and the formula barely changes. First comes a safety net you actually believe: regression tests so you do not blow up core functionality, plus a clear ranking of mistakes. For a database company, writing data badly can take months to unwind, so that class of error is catastrophic. At Google he built auto-experiment frameworks: every change ran on one machine for 15 minutes to confirm it did not crash, then rolled to 0.1%, 1%, and 10% with measurement the whole way. Once that exists, engineers design experiments instead of fearing launches.

Second is inner-loop productivity—how fast a single change can land—because that is what decides how many changes you get through. Snowflake, starting about two years before he arrived, rebuilt the unified product to be extensible so that unity would not become a speed tax. AI features rode that architecture. Third is leadership clarity. Every team faces infinite work, so he forces the AI groups to make a three-month commitment and a two-week one, then calibrates against what they actually shipped. "Life boils down to say what you're going to do and do what you said you would do." Quality and speed are both requirements, and customer usage of new products feeds the next round of momentum.

Asked what he is proudest of in AI so far, he names Cortex Analyst as probably the hardest product Snowflake has designed and launched. Cortex AI is "predictable infrastructure work" underneath—inference optimization, the ugly fight for GPUs—but Analyst and Document AI apply the company's strengths to new ground. Polaris, the cloud catalog, shipped in about three months. The deeper belief is that "virtuosity trumps strategy all day long": execution speed and the speed of reacting will beat a fixed plan in a world where you cannot know which product explodes or what GPT-5 will do. He cites a Napoleonic line that translates as "I commit and I adapt." Grady notes the continuity with Slootman, who liked Patton's line that a good plan executed violently today beats a perfect plan tomorrow.

The misconception he wants to kill is that Snowflake does not really get AI. Early on the company leaned on partnerships. His later realization was that AI is a platform change: a new way everyone will get to software and applications. Once that clicked, the product consequences followed. AI had to sit at the center; Snowflake had to make building easy and had to build the most important applications itself. Cortex Analyst is a direct-to-business-user product of a kind the company had not really shipped before. Combined with Iceberg, Polaris, and a team that runs from foundation models through inference to product engineers, he thinks that position is a strong one.

Zooming out, Ramaswamy hears a collective belief that a GPT-5 is on the horizon, but nobody has a clear bar for what it would represent. GPT-4o was faster and folded in native multimodality in a way he finds impressive, yet reasoning and the ability to form plans did not feel like a step change. Agents are hot, but until Cortex Analyst, people did not really believe you could build a reliable talk-to-your-data app. The bar for business users is brutal: 75% accuracy is one answer in four wrong. If the next models jump in multi-step reasoning, they will unlock a class of applications we cannot picture yet. If training a great model starts requiring a billion dollars, he hopes we do not go there, because it shrinks who can play, and competition is healthy.

Grady describes a pattern every AI builder knows: the first two weeks feel like magic, then months of corner cases and missed accuracy leave people disillusioned. If today's foundation-model capabilities were frozen, what would still change? Ramaswamy's diagnosis is that ChatGPT quietly trained us to treat the model as omniscient. Google Search never tells you a query is dumb; it just returns ten pages. Some of that is enthusiasm, and some of it is "plain dumb." You should not take raw model output to the bank. Grounded chatbots should become the norm: every answer should say where it came from, and verification should be easy. You need a test framework and ongoing observability, because these models "react pretty violently to the addition of a period in a prompt."

Snowflake's pitch is that this is the company's core strength: a predictable success rate for a chatbot, an agent-like app, or Cortex Analyst in your domain, so you feel good deploying it. Even if GPT-5 never happened, there is a lot of magic to be done—but it is also just work. Grady quotes the line that people miss most great opportunities because they are wearing coveralls and look like work. Any ML engineer worth their salt will show you an AUC curve, which is the trade-off between how hard you squeeze the model and how good it is. There is no perfect answer. The more teams treat AI applications as having that curve—reliability versus the ability to respond—the more value they will actually land.

Huang asks whether AlphaGo-style tree search at inference time—the idea behind systems like Cognition's Devin—is the path to general reasoning. Ramaswamy is cautious, because those methods depend on a model that can grade a position in state space, which AlphaGo can do for a Go board and open-ended plans often cannot. Cooking looks simple until ten ingredients and twenty variable steps explode combinatorially, while only a couple of recipes are actually perfect. Structured diagnosis is a better fit: at Google, a daily "Bean Counters" email would go sideways every few weeks until an analyst drilled through slices and reported that it was Easter in Germany and Ascension Day in Brazil, a class of real-world wrinkles that took a decade to model into forecasts. An "AI analyst" prototype that calls Cortex Analyst, reads the output, and suggests the next cut already removes 60 to 70% of that labor.

On consumer AI search, Ramaswamy is gracious about Perplexity: right time, right place. Neeva had converged on a very similar product two or three years earlier, and "timing ends up being everything." Nobody wants to hunt through an eight-page document for two lines they care about; that instinct is correct. The harder point is the business. Search is carefully preserved with business contracts, not with consumer choice. Consumer choice, he says, is "fiction" in a lot of what we do: we eat what is put in front of us, and we search with the default engine that came in the browser. A few players control the entry points. Having the coolest product is not the same as changing that behavior.

On incumbents versus startups he takes two lenses. On models, open-source weights plus a player like Meta with "infinite budgets" make training from scratch look very hard unless you have an attached hyperscaler or an attached business. He hopes the world does not end up with three GPT-5-class models. OpenAI, in his view, has created a solid product, not merely a foundation model. Historically, each generation of Silicon Valley companies learns from the last and leans in earlier. When he was at Google, Alexa looked like it would take over search via the home; Google "put a pile of money into it and nothing came of it," and it did not matter, because the cost of a disruption is way higher than the investment required to defend. Incumbents know what can be disrupted.

He thinks of AI as shuffling the tiles on enterprise software, and Snowflake intends to lead the charge rather than wait. Every enterprise software company is leaning in the same way. For startups he would feel much safer creating a new category—mass-scale image generation, video, voice, specialized marketing—than a story that I can do XYZ faster because I am AI-enabled. That latter pitch, he says, does not have a whole lot of legs.

Asked whether ChatGPT can become the next Google, he points first at the phone as a controlled environment with enormous consumer potential. Something as ridiculous as copying an address from email into Uber is still "so dumb, so hard"; Siri still cannot do it. Apple can mandate that apps expose functionality through language-model APIs or risk getting no traffic—"a pretty good incentive for everybody to get in line." He calls ChatGPT a stunning product experience that has turned "visually illiterate people like me into budding artists." Learning Hindi, he typed a prompt asking for a CSV that maps number strings into Hindi for Quizlet; it was faster to type than to describe, and the file arrived in ten seconds. Work he used to do with Python scripts on structured data he now does in English. Is it a great product and a great business? Yes. Being the king of search, he adds, is a few more zeros, and they do not come easy.

In rapid-fire, Ramaswamy says he most admires the people doing foundation-model work "on the cheap," without infinite resources—researchers such as Danny Yogatama at Reka, and colleagues on Snowflake's own team who keep shipping under a limited budget. The doers imagining the future under tight constraints are the ones he wants to celebrate. His favorite AI application, by far, is ChatGPT, for the daily utility. The app he wishes existed is a real talk-to-your-phone assistant that can mediate between apps, because flipping between them for tiny tasks is still such a pain.

The best thing that could happen over the next decade, in his view, is not a pile of fancy new techniques. Software has been encoding human thinking and action for fifty-plus years. AI as an enabler of access—both to creating software and to using it—for far more of the world's people would be a genuine step up. He is proud that Google Search, via Android, reached almost every human on the planet. AI models as the new layer between humans and software, and between software and software, would make that functionality vastly more accessible. Newer models can unlock newer classes of apps, but the starting point is plainer: get the answers right, show where they came from, and keep the promises you made to the people using them.