A 16K Context Window Holds About 350 Rows. That's Why Tables Need an Agent.

Nathan Labenz introduces Alex Watson as founder and chief product officer of Gretel AI, a synthetic-data platform for developers that has been generating tables since 2020. The name, Watson says, is a nod to digital breadcrumbs, Hansel and Gretel style. The original vision was to make data that cannot be shared, shareable: hospitals exchanging records, research labs and financial firms collaborating, without compromising anyone's privacy. Three years in, the product has grown past that founding story, but privacy is still visible in the differentially private training they ship.

Watson's previous company was Harvest AI, a security startup that scanned the cloud for important data. AWS acquired it in 2016, and he spent about four years as a general manager for Amazon Macie, the service that finds personal data in AWS. Even Pinterest and Airbnb, he says, struggled to enable internal access. AWS, by contrast, had a 500-person compliance team that could work wonders. The gap was tools. Macie launched to 6,000 customers in its first week and had to run named-entity recognition at petabyte scale, a mix of classic NER and regular expressions that needed retuning whenever the data changed. That is one reason he is grateful for transformers: natural-language questions can now make those changes without rewriting the Swiss Army knife.

Gretel's first open-source drop went out in 2020, about a week before the pandemic. There were no transformers in the stack yet. They used an LSTM and partnered with Google's TensorFlow team on DPSGD, training with differential privacy so the model would not memorize secrets. The first real experiment was whether a language model could stop learning text and start recreating the statistical distributions inside a data set. Researchers at UC Irvine had thousands of patients and only tens to twenty with a rare disease. Gretel boosted the minority class with new labeled examples they could not go back and collect. Downstream detection improved. Fast-forward, and Watson says the business is about 50/50: safe sharing with up to mathematically provable privacy guarantees, and improving machine-learning sets for fraud detection, ad recommendation, even text. The guiding line is blunt. Customer data is messy, it has gaps, new examples are too expensive or impossible to recapture. The company exists to help you build better data, more accurate or more private than what you started with.

Labenz has done enough analysis to know why Rocket Mortgage will not let you copy the crown-jewel customer file onto a laptop. He also needs a theory before he will run pivot tables on something fake. Watson's answer starts with the loss function, which does not, by itself, tell you whether the model captured the real-world distributions you care about. Regardless of modality — text, tables, time series — they train until the model can recreate data matching the original distribution. Then you can alter that distribution for the task. At the end of training they sample a one-to-one equivalent of the real set and, from a statistical perspective, throw the kitchen sink at it.

The synthetic quality score is a 1-to-100 composite: pairwise correlations, per-field distributions, even PCA distances between real and synthetic. Above 80, they expect it to work for the machine-learning jobs most customers want. Below that, it might be fine for software testing; Labenz would still not build a pivot table on it. A second check is task-specific. After training, the platform can automatically run a regression or classification job, the thing customers were already doing in Vertex or SageMaker, so not everyone has to write the extra code. Objective score plus a test of the actual downstream task is how they try to manufacture confidence.

On the generation side there is another insurance policy they uncreatively call validators. After the model emits a row, a second check asks how far it sits from the training distribution and filters the outliers. In tables, that is how you avoid inventing a 135-year-old. In text, it catches the model when it goes on a rant. Nobody can inspect a million records by hand, so quality control has to be automatic. Labenz notes a conceptual split people miss: the model predicts a distribution, sampling produces the row, and Gretel evaluates at row level, not token by token. Watson says that is a byproduct of starting with tables. They have used LSTMs, GANs, diffusion, and now transformers, and they always look at the whole record.

Gretel was about to release what they call a tabular LM: an agent that plans and executes, so people can work with tables in natural language. At the core, the agent decides whether to use an LLM to generate data or to write code with one of their tools. The language model itself was fine-tuned on data sets crawled from the internet, GitHub and Kaggle with open-source licenses. They got lucky with machine-learning papers that link data sets in the README, license included. The point was to show a generation model what good data looks like. OpenAI, PaLM, and Llama models have seen most of the public web. They have not, Watson says, been trained largely on tables. A 16K context window, at fifty tokens per row, holds about 350 rows. Most working data sets are orders of magnitude bigger. GPT-4's idea of a table is usually a Wikipedia page of a couple hundred rows. It has not learned that relationships across thousands or hundreds of thousands of rows can matter.

That is why no LLM today can generate a 100,000- or million-row data set, or walk into your warehouse and edit it, in one pass. Adding fields, summarizing product reviews, hunting anomalies, converting Celsius to Fahrenheit across the whole store: all of that is bigger than a single batch. The agent's first job is to interpret a query like "create a demo set with a November sales spike, a million rows" and chop it into problems the generation model can finish one window at a time. The analogy is asking GPT-4 to write a book. You get a short essay. If you plan an introduction, a character arc, and a conclusion, chapter by chapter, you get a novel. They sample as many tokens as the model can handle, then evaluate row by row. Efficiency drops if the model never learned that more than a couple thousand tokens are relevant.

The other job of the agent is to notice when the LLM should not be in the loop at all. An incrementing ID or a Fahrenheit conversion is a calculation. The model will approximate; you want the real answer. Summarizing whether a review is positive or negative is language. Mix them in one table, as HR files mix doctor's notes with intake observations, and the agent should route compute to code and semantics to the model, then stitch the columns back together. For the first tabular LM they treated numbers as text rather than encoding distributions with anything clever. Watson thinks a more curated, diverse set of high-quality examples is the next move, the same lesson other labs are learning, plus longer context so the model can actually see those cross-row relationships.

Labenz wants to patch his own product, a video-script model that has to handle a sparse, international long tail. Fine-tuning GPT-3.5 on a few hundred samples, he finds that one extra example is not enough and five to ten high-quality ones, aimed at a weakness, often is. The problem is inventing those extras. He does not know the structure of the French postal system. Making them up is garbage in. Asking GPT-4 is worse in a different way. RLHF-tuned models mode-collapse: ask for a random number and 42 and 97 show up far too often. He would not trust that distribution for representativeness, and he thinks OpenAI would agree the model was trained for a different job. That, to him, is Gretel's distinct position in the market: not another assistant.

Watson has used RLHF too, including to stop models from returning PII. The interesting flip is that enterprises sometimes need the opposite of consumer manners. Customers generate spear-phishing emails, no joke, to test their own detectors. Healthcare companies trying to augment patient records get refused because the assistant thinks the request is harmful. Off-the-shelf models also collapse toward one kind of name, address, and United-States-average demographic when you ask for mock data. Gretel has been training the model to return more diverse, better-aligned demographics for a given zip code. Sometimes you want real data. Sometimes you want ethically aligned data. RLHF is a tool. What matters is which way you point the loss.

Labenz wonders why you cannot just slide temperature: zero for the mode, one for the real distribution, two to over-represent minorities. Watson says that even a high temperature will not, on a badly imbalanced set, surface a 1 percent class often enough. You need an explicit training signal. Pinterest's engineering blog, which he loves, showed wedding-ring search returning hands of many skin colors, a feature you do not always want. Ad recommendation is the other extreme. A major social-media customer, hit by the death of third-party tracking, faces a thousand impressions and one or two clicks. That minority is the whole job. You have to tell the model those rows matter and ask it for meaningful new variants. Diversity on, diversity off, and rare-class boosting are different knobs. Temperature is not enough of a knob.

Labenz treats a lab's ability to control a model as a canary: if you cannot stop it being offensive today, can you stop a bioweapon prompt tomorrow? Watson is optimistic about AI and sees synthetic data as a tool someone could use for alignment or for harm. He wants warning flags everywhere and, as far as possible, user control. He prefers an open model that can be adapted over a closed world in which a few companies decide what may be returned. The White House executive order that week, and the UK AI safety summit, are signs that the right topics are getting attention. Gretel has about 65 people. He does not want regulation written for firms that already have 100-person compliance teams to squeeze out the ones that do not. Labenz's jab is that the hundred-plus-page order mostly orders another ten thousand pages of reports. He also thinks the current game board is better than it could be: the leading labs are serious-minded, and even the most rogue large company is more responsible than a true don't-care would be.

Gretel's pretraining is on the order of 500 billion tokens. Labenz's intelligence puts GPT-4 around 10 trillion, twenty times the tokens and far more parameters, three orders of magnitude in compute. Watson is not trying to compete at that scale. He points at Microsoft's *Textbooks Are All You Need* result: a 1-billion-parameter model, cheap to train and fast at inference, trained on a relatively small, diverse set of examples, doing work far above its size. Life sciences, fintech, video companies can all ship small, unique models the giants cannot match at the same level. Gretel's own split is the same idea. A large model parses intent. The small, fast one generates the data. You do not need 10 trillion tokens to enter the market.

Labenz worries most about super-general systems. AlphaFold and AlphaGo are easier to keep in a box because they do one thing. Eric Drexler's *Comprehensive AI Services* manuscript, already about five years old, sketched an ecology of narrow superhuman tools so you might never need an uncontrollable general mind. The temptation, meanwhile, is to find out what happens at 10^27, 10^28, 10^29. A 50,000-H100 cluster running a hundred days toward 10^30 is small-city electricity, which means a state can still intervene, until algorithms make that impossible. Watson's analogy is the brain: nature stopped growing a bigger blob and started specializing parts. He wants a world of mild expert models, trained on small domain-specific sets, orchestrated by a large model that is merely smart enough, auditable, understandable, not an Uber-intelligence nobody can inspect. Labenz agrees, with more fear in the mix. Some leading developers, he says, still treat building the most powerful thing they can as the point, safely if possible. That is the part that does not seem wise, and the part a state can still slow.

Asked whether language models are stochastic parrots or reasoning engines, Watson declines the metaphysics. The north star is the data scientist's actual problem: messy data, gaps, examples you cannot create, collection you cannot rerun. Better data, more accurate or more private, is the guiding light. They ship early, take feedback, and decide how much to invest in a better agent or a better million-row time-series stack based on what users do. Retrieval, ReAct-style decomposition, and the training itself are all dials. Most organizations, he says, have 80 to 85 percent of their data in some structured or semi-structured form. That is a niche that is also almost the whole warehouse. OpenAI and Anthropic will show up. For now the combination of advancing the LLM only as far as needed, plus the rest of the toolbox, is working.

Labenz tries to dissolve the parrot-versus-engine fight inside Gretel's own architecture. The generation model is supposed to be a high-fidelity stochastic parrot; the randomness is the product. The planner has to be reliable. Maybe the production-possibility frontier splits, and Gretel is pioneering the honest-random side while others push reasoning. Watson likes the idea. It matches the strangest customer conversation of his career, at an HL7 conference devoted to synthetic data. FHIR is the dominant medical-record format. In that world, MITRE's open-source Synthea has spent four or five years generating purely statistical patient records for testing. The complaint, the opposite of every other customer he has ever had, was that Synthea is too clean for machine learning. Real data has fuzz. You want a little stochasticity, not crazy, and you have to reason about 50,000 or a million examples at once, not one at a time. Starting with tables, where right and wrong are easy to see, made Gretel think about that earlier than labs now drowning in beautiful generated text and images.

The working rule is: let the network generate, then examine every step and filter what walks off the reservation. Labenz hears the old computer-vision trick of training on systematic corruptions, noise, distortion, cropped aspect ratios, so the model survives the wild. Medical records have their own mess: illegible, incomplete, contradictory, two patients in the same hospital with the same name and birth date. Synthetic data that is too tidy will not prepare you for that. Demo data and UI mock data, Watson notes, are still where a lot of people start, names and addresses and protected-class fields included. That use is mature. The harder job is keeping the dirt without keeping the patient.

Labenz fine-tuned an early OpenAI interface on his own writing and résumé, cranked the epochs, and never got it to answer as Nathan Labenz. Jeremy Howard says an LLM can memorize from one example. Watson's setting is different. When they train language models from random weights on a customer's set, the model has a very high propensity to memorize and replay secrets. A 2020 Berkeley paper, *The Secret Sharer*, from Dawn Song's group, measured how fast even rare occurrences get stored and how likely they are to be played back. Small fine-tunes on a huge pretrained corpus are harder to detect, because you may be touching only a slice of the weights, but the risk is still there. Customers run canary tests after fine-tuning, trying to make the model autocomplete a credit card number.

Gretel's first layer is simply deleting PII, names and emails, with an LLM or any NER. The trap that still trips people, especially on medical tables, is combinations. Remove the name and you still have height, zip code, and a disease. None is identifying alone; together they are. Writing style can do the same in text. The answer on both tables and text is differential privacy: inject a quantified amount of noise into the optimizer and clip gradients so a rare string like "my name is Alex, I'm six feet, I live in Southern California" cannot be memorized and replayed. You get a guarantee per example or per person, not a vibe that you could not extract it today. That has opened doors. National-level healthcare organizations have gotten approval to share synthetic medical records between hospitals, not merely de-identified files with the names stripped. You want the model to learn about a disease, not about a patient.

The cost is utility, especially on small sets. Noise is a large fraction of a tiny file; Gretel's experience is that 100,000-plus examples is where it starts to make sense, which is why the Census Bureau and Google and Apple's next-word and emoji prediction use it at scale. New work, including talks at a Gretel conference from Google researchers, says you can claw the accuracy back with much larger batch sizes, more compute, and interleaved public examples during private fine-tunes. Privacy without a utility hit becomes a budget question. For regulated industries that have to convince a supervisor that a patient in the set cannot be recovered, that is the key.

Labenz closes on the scary paper titles: self-consuming generative models go mad, generation after generation, until things get weird. Is synthetic data a long-term problem, or are the weirdnesses just undialed details? Watson is firmly in the details camp. LLMs today often emit a cleaner, more diverse version of what they were fed, which is why people run email through them and use Grammarly. *Textbooks Are All You Need* is the hopeful exhibit. As long as the synthetic data is high quality and ideally better than the seed, he does not see an inevitable march to mode collapse. His joke, and the last word: as long as the next generation of models does not all start with "I'm a helpful AI assistant, how can I help you," they are probably pointed the right way.