When GPT-4 Writes SQL, Four Tables Is the Breaking Point

Nathan Labenz opens with a pattern he has seen across established software companies. They are not especially afraid of AI-native startups, because the moat sits in private data and messy business context, but they are rolling up their sleeves to put models into the product. His guest is Juan Sequeda, principal scientist and head of the AI Lab at data.world, a company that spent years building a data catalog before ChatGPT arrived. Sequeda's short analogy is a library card catalog: hundreds of databases, tens of thousands of tables, and a pile of dashboards all need a place that records what exists, who uses it, and where it came from.

The company started around 2016 as a "GitHub for data," an open catalog of public datasets. That product still exists, Sequeda says, as the world's largest open data catalog: more than two million users, about half a million datasets, and roughly two-thirds of Fortune 500 companies on the platform. During COVID, global pandemic data lived there in the open. data.world is also a public benefit corporation, with a charter that includes advocating for open data and linked-data standards and serving as an archive for the world's data. Around 2019 the same platform, already proven at web scale, became the enterprise catalog and marketplace business.

The technical bet was older than that. From day zero the stack was a knowledge graph built on open web standards: RDF for the metadata model, OWL for ontologies, and SPARQL for queries. The moment a customer's sources are connected, collectors pull technical metadata and weave it into a graph automatically. Databases hang off tables, tables hang off columns, dashboards hang off the queries that feed them, people hang off the assets they created, and policies hang off tables that contain personal information. Sequeda's first application for a knowledge graph, he likes to say, is simply understanding the infrastructure you already have.

Customers mainly use that graph for three jobs. Search and discovery is the one data scientists feel first: instead of wandering a data swamp, they get something closer to Google's knowledge panel, with related assets appearing beside the hit. Governance is the second: who can use which data, and where the sensitive columns live. Data operations is the third, especially lineage, so a broken dashboard can be walked upstream to a table or a pipeline, and a change can be checked for blast radius. The AI argument sits on top of all three. Foundation models were not trained on a company's private meaning, Sequeda says, and data is the bloodline. The catalog is where semantics, context, and knowledge live, which is exactly what the models lack.

A knowledge graph is not only a picture of technical metadata. Sequeda draws a second bubble-and-line sketch for any business domain: in e-commerce, customers place orders, orders contain lines, lines point at products, and shipments go to addresses. Data to fill that picture can arrive from relational databases, JSON feeds, APIs, or plain text. The graph is a way to integrate all of it under one model. What takes time, he says, is the knowledge-engineering process: sitting with people until you know what they mean by customer, order, or active order.

Ask five people and you will get five answers. Sequeda's standing line is that if the humans cannot agree, the machines certainly will not know. Large language models can now propose candidate models and, through a chatbot, pull definitions out of employees' heads. Governance is the step that follows: someone still has to pick a definition and stand behind it. That human bottleneck is why so much "tribal knowledge," as Labenz calls it, never makes it into systems, and why generic agents feel alien inside a company they have never worked for.

The easy demonstrations of "chat with your data," Sequeda argues, have been on unstructured files, PDFs, and images that never lived in a database at all. That is where vector stores and chunking earn their keep. Structured data is harder and, he thinks, more valuable. It is the thousands of tables and tens of thousands of columns behind CRMs, ERPs, reports, and board packs, and executives expect those answers to be accurate and explainable. Text gives you more freedom. The numbers that run the company do not.

Labenz remembers a junior consulting project at Washington Mutual, then the country's largest thrift, in the mortgage-crisis years. The data was a swamp of derived columns with suffixes that meant nothing to him; he finished an analysis, someone told him it was totally wrong, and he had no way to have known. Sequeda's reaction is recognition rather than surprise. The problems companies describe today, he says, are the same ones people described ten, twenty, and thirty years ago, back to the warehouse era of the 1990s. The cloud made work easier and self-service more common. The disease did not change.

At bottom, he says, companies use data to make money, save money, or reduce risk. The most primitive way to get an answer is still the spreadsheet: you ask someone for orders or latest customers, they send an Excel file, and you VLOOKUP your way to a conclusion, then repeat it next month. The hidden failure is semantic. When you say "orders," you have no idea whether the person who pulled the file heard the same thing. That is how enterprises have lived with data for a very long time.

The next step is the query approach: skip the monthly dump and hand over database access. A 15-line query picks up a join and becomes 30 lines, then 100. Sequeda has seen queries ten to fifteen pages long that take twenty minutes to run, with the result dropped straight into a report for the CEO. Those queries are packed with knowledge that nobody can inventory. Everyone is editing their own copy, so a change in one person's logic never reaches the next. Billion-dollar decisions, he says, rest on that pile.

The promised fix is a standard warehouse: millions of dollars and several years to align everything. When the new system finally answers the old question, the number rarely matches the one people have been computing themselves. They trust the process they control. Warehouses fail for social reasons, not technical ones. Then the industry flipped to ELT, dumped data into lakes, and watched the lakes become swamps. Through every generation, Sequeda says, nobody treated meaning as something to manage. LLMs are now making that gap impossible to ignore: if you want to trust an answer, you have to know what the words mean.

Sequeda splits the profession into producers and consumers. Producers live on the backend: they move data, migrate it, and keep it alive. Consumers search for it, build dashboards, train models, and answer questions. On the consumer side the 80/20 rule is folklore, 80 percent of the time spent cleaning data, and people shrug it off as "data janitorial work." That phrasing infuriates him. Cleaning is how you learn the critical meaning of the business. Treating it as an annoying click-script is how organizations refuse to make meaning a first-class citizen.

The gap between the two sides is structural. Consumers speak in business language, customers, average order value, named metrics, while producers hear tables and columns and have to guess which shape of the data is the correct one. Sequeda thinks a missing role is filling that aisle: a knowledge engineer, knowledge scientist, or data product manager who can sit with a non-technical person, draw the model on a whiteboard, and then walk it over to engineering. People already do fragments of that work. Almost nobody owns it.

His consumer analogy is Amazon. Search for a water bottle and you trust the listing with photos, descriptions, reviews, and stars, not the one with a blank page. Someone had to put that metadata in. A catalog should feel the same: producers stock the shelves, a product manager decides what gets a beautiful data-product treatment, and consumers can search without wandering. data.world, he adds, is unusual in letting you find the data and then query it in place through virtualization, so Snowflake or Databricks tables never have to move. He also wants the graph to store questions and answers themselves: who asked, what query an LLM generated, which steward stamped the official number, so the next similar question can reuse a governed result.

Labenz asks the uncomfortable follow-up: how accurate are the humans we already trust? Sequeda has watched critics dismiss a 60 percent machine score without ever measuring their own process. He once reverse-engineered a report a customer had used for years and found a hardcoded `0.02` multiplier, a special discount from two years earlier that the author had forgotten. Every number since had been wrong. The honest coda, he says, is that the company still made money, which is why "accurate enough" is a cultural question most organizations have never asked. Labenz offers his own version: a month of podcast downloads spiked, the next month crashed, and the team hunted for a content problem until they noticed they had published nine episodes, then seven. Sequeda's habits of mind are blunt. Ask why five times, tie the work to quarterly goals, and run an "iron thread": one thin path from question to outcome before you try to boil the ocean.

Text-to-SQL is not a new hobby. Computer scientists have worked on it for thirty years, and academic benchmarks now advertise 95 percent accuracy. Sequeda does not believe those numbers transfer. The schemas are tiny, the semantics sit in the table names, and the questions look like a laundry list disconnected from how a company actually works. Enterprise reality is thousands of tables and tens of thousands of columns. At Snowflake Summit in June, a group of product managers challenged him to stop talking and publish a benchmark. He got off the plane and started.

The schema is an open property-casualty insurance model from the Object Management Group, chosen because it looks like a real enterprise domain rather than a toy shop. Questions sit on two axes. Business difficulty runs from "show me all open claims" to questions that need strategy, aggregation, and math. Technical difficulty runs from two or three tables to eight or nine joins. Cross those axes and you get four quadrants, so you can see where a method holds up and where it falls apart. That framework, Sequeda says, is half the contribution: other people can reuse the quadrants on their own data.

The third ingredient is an explicit context layer. A claim has a claim number, and there is a `claim` table, which looks reassuring until you notice the table also has a column called `claim identifier`. That is not the claim number. The real one is `company claim number`. Policy holders are rows where a role column equals `PH`; agents are `AG`. None of that is obvious from DDL alone, and all of it is the kind of mapping a business person carries in their head. The benchmark writes those mappings down as an ontology and feeds them to the model.

The setup was deliberately bare. They used the 8K-context GPT-4, pasted either a SQL DDL or an OWL ontology into the prompt, and asked for a SQL query or a SPARQL query. Across the full question set, the knowledge-graph condition produced accurate queries about three times as often, landing around 60 percent. In the high-complexity quadrant, any question that required joining more than four tables caused GPT-4's SQL to fail every time. Sequeda presents that as a low baseline for the community to beat with better prompting, retrieval, and context, not as a finished product. For companies that want their own evals, his recipe is the same three checks: you already have data; catalog the real questions, who asks them, why, and how they get answered today, then drop them into the quadrants; and if you have no catalog, start there, because the model will never invent your private semantics.

Sixty percent across all questions is not good enough to put in front of a decision maker, and Sequeda does not pretend otherwise. The next design, the one his lab is building, starts by asking whether the question is answerable at all. Extract the concepts with a model, look them up in the graph, and if something is missing, refuse with a reason. Users then see either an answer plus an explanation or a clear no. Knowing what you know and what you do not know is itself a source of trust. On the subset you can actually answer, he suspects accuracy can reach about 90 percent.

The distinction he wants people to remember is simple. If the LLM answers the question, the number is forever probabilistic. If the LLM writes code, a deterministic system executes it. You can reason about code, govern it, store it, and use it as an explanation. The query runs against data whose lineage, owners, and authorizations already live in the catalog, so you can say not only what the number is but which table it came from and who signed off. A bank-account balance, he notes, had better not be a sample from a language model.

Explanation is not one artifact. A business user wants a sentence in English and a name to call; dump SQL on them and you have failed. An engineer wants the query, and a fluffy summary will get called out as nonsense. Because the knowledge graph can be traversed to any depth, the same answer can be rendered at either altitude. Fine-tuning, in this picture, should hit metadata rather than data: billions of changing ledger rows are expensive to extract, stale immediately, and still leave you with a probabilistic answer. Fine-tune on patterns of questions and queries instead, then run static analysis on the generated code as post-processing.

Labenz's own experience lines up. Fine-tuning GPT-3.5 on chain-of-thought traces, sometimes with as few as a hundred examples, changed how the model worked through a problem, but it did not teach facts; table definitions still had to sit in context. Sequeda agrees: behavior can be tuned, knowledge belongs in the graph. All of these pieces, prompt, vectors, metadata fine-tunes, checkers, will hang off an agent framework with many states and tools. Today's RAG pipeline, retrieve then generate, is in his view the simplest and most naive agent system. Planning and agents are fifty-year-old AI, and he would rather the field stand on those shoulders than reinvent the wheel. The lasting IP, he thinks, will be how those frameworks are designed.

Labenz brings a live problem from Athena, the executive-assistant company. Client knowledge is almost all unstructured: onboarding interviews, transcripts, summaries, chat logs, with no schema at all. The team wants a long-lived profile that updates as preferences change, including a way for new statements to supersede old ones. Retrieval today is a vector store plus HyDE, hypothetical document embeddings, which helps, but they do not yet have a maintenance story. Sequeda's answer is not to accept the premise. There is always a schema, he says. It is just implicit in the concepts people keep talking about under different words.

His practical move is almost embarrassingly small. Dump the text into GPT and ask it to generate a business glossary, taxonomy, or ontology, then have humans correct it. Once that controlled vocabulary exists, it can shape retrieval, chain-of-thought, and even the UI. He has also watched a side effect: people start using the official words because those words actually return answers, and the organization grows a shared language. The schema was there. Someone finally wrote it down.

Architecturally he expects a mix of vector indexes and graphs, not a religious choice of database. Labenz mentions the Stanford generative-agents paper, in which characters periodically roll raw observations into higher-level memories. That is a natural graph: a summary node with edges back to the observations that produced it. For enterprise warehouses, Sequeda is equally firm about what not to do. Do not copy Snowflake or Databricks into a graph database. Use a virtual knowledge graph: query the graph, translate to SQL, run it where the data already lives. That is how their own benchmark ran.

On the vendor landscape he is blunt. Vector databases are a feature, not a category. Every SQL database and every graph database is adding vector search, if it did not already have it yesterday. Independent vector stores will need a true scale niche; otherwise the database you already run will be enough. He expects one winner in that niche, the way MongoDB was the one durable winner of the NoSQL wave, and the rest to shrink. Snowflake, Databricks, and the clouds will also absorb unstructured workloads rather than leave them on the table. Labenz notes that Chroma's Anton had argued the opposite, that never-before-stored data will grow the whole market tenfold. Sequeda's reply is that the incumbents will simply support that data too, and that adding a SQL interface to a vector-native store is not the same as already being the system of record.

Question answering is not the only job. Sequeda is already using models to draft metadata so nobody starts from a blank page, and to scan a schema, without touching the rows, for columns that probably hold personal information. With WPP, described here as the world's largest advertising group, the catalog proposes questions you could ask of the data you already have. People look at the list and say they had not thought of that angle. Search gets more natural-language. Code gets explained. The through-line is productivity: less empty-page work, more time on judgment.

The use he is most excited about is knowledge acquisition. Meaning has always been trapped in conversations. Now a GPT can play a Socratic interviewer, asking what "active customer" means and why that question matters, then asking why again. In a day you can talk to twenty or thirty people and have the model summarize the competing definitions. The same pattern maps how someone actually does their job, which is how you see the manual steps that operational excellence is supposed to remove. For the first time, Sequeda says, pulling knowledge out of heads can scale.

On jobs he does not soften the sentence. Almost everyone sits in some supply chain that AI will touch. Writing code, especially the curriculum of coding bootcamps, will be heavily automated; the scarce skill is the algorithmist who can decompose a problem so that generated code can be 80 or 90 percent right. Generic analysts who only report well-known industry metrics will be replaced, because the models already know those metrics. People who connect unusual dots, and people who will decide that "active customer" means these three things for these reasons, will not. He even expects universal basic income to sit near the top of a U.S. presidential race eight years out, and "served by a human" to become a premium. Eureka-style scientific discovery, Labenz adds, still needs scaffolding; the models are scraping the bottom of that threshold, not clearing it alone.

The closing architecture is the line he opened with. Foundation models are experts in language and general knowledge. They do not know your organization, and you should not want them to memorize it. The knowledge graph is the internal brain: accurate, governed, explainable. The work of the next few years is the integration point, starting with naive prompting and RAG and moving toward plug-and-play, the way companies already refuse to bind themselves to a single cloud. Invest in that brain, Sequeda says, because it is the private piece a model cannot generate for you. Then keep a list of the questions the business actually asks, put semantics on equal footing with pipelines, and pull one iron thread all the way through before you scale.