GPT-4 still writes SQL that will not run. She put a guardrail on the model

Nathan Labenz opens The Cognitive Revolution by introducing Shreya Rajpal as a former machine-learning engineer at Apple, a founding engineer at Predibase, and the creator of Guardrails AI. The Python library adds a layer of output validation and correction between language models and ordinary deterministic code. Anyone building AI products over the last two years, he says, already knows the pain: models do not always follow instructions, and sometimes they go entirely off the rails. GPT-4 and Claude v1.3 follow instructions far more reliably than earlier models, but that extra capability has only expanded what developers try to ship, so the reliability problem remains both critical and ubiquitous.

He invited her almost the moment he got off a previous recording with Matt Welsh, CEO of Fixie.ai, who had mentioned the project. Rajpal says she was solving her own pain. At the end of 2022 she was tinkering on the usual Twitter demos — chatting over proprietary documents — and could see the potential while testing as a developer. The experience was not reliably the one she wanted. The models are potent, she says, and also inherently stochastic and hard to control.

Unlike traditional machine learning, the developer has not trained this model, so you cannot throw more data at your use case. The only knob is the prompt. The folk solution is more verbiage and more exclamation marks to make the model listen, which she calls woefully inadequate. Guardrails is a specification framework: you know what the right output looks like, you decompose it, you validate each component, and if a quality criterion fails you get an extensible set of tools to address it.

The other inspiration is years in autonomous systems. At a self-driving company, a powerful deep-learning perception model often feeds a more rule-based decision system. The job is to keep the interface between that stochastic system and that deterministic system robust, not brittle, when perception has a bad day. She wanted the same idea for language models, built in an extensible way rather than a domain-specific one. By the time of recording, the GitHub project had taken its own little rocket-ship ride to about 1,200 stars.

Labenz asks for concrete failures, and Rajpal starts with her own Chase credit-card agreement. The task is to extract key terms into a nice JSON object. As a user she already knows the constraints: an interest rate must be a number, it should carry a percentage sign, and it should fall in a reasonable range. A fee name should be concise. If that JSON is headed for a downstream data sink, an unreproducible language-model extract is a bad colleague. Guardrails lets her impose those constraints on each extracted entity, including a description that has to be a certain length and actually relevant to the entity it travels with.

Her favorite example since then is text-to-SQL, a wildly different domain with the same idea. Users want to ask natural-language questions over their data and get a query they can execute. Correctness, defined in advance, is harsh: the SQL must actually work on your database; some tables are private and a customer should never be able to query them; DROP, UPDATE, and INSERT predicates should be filtered out. As you set up the task, you add all of those constraints.

The library takes the database schema and stands up a SQL sandbox. Every generated query is executed there. If it is not executable, Guardrails wraps the errors, sends them back to the large language model, and asks it to correct itself until something actually works on that specific database. You can still layer on the table and predicate filters. The through-line, she says, is that the developer has domain expertise for what correctness means and how much they care. If a bad query is totally useless, you handle it one way; if you just want to know and will deal with it later, you handle it another. That is the product: implement that judgment, instead of rolling the dice.

Labenz maps a spectrum of errors. At one end are the failures developers already know from code: syntax errors, a variable that does not exist, anything a traditional toolchain can catch. At the other end is a domain of correctness that code has never really accessed: is this a reasonable approach, a faithful summary, a translation without gaps in logic? Guardrails covers that mix in the validators, and he wants her to walk the space.

The exciting part of the library, she says, is that the framework works even where you cannot use an assert statement. You can mix large language models, rule-based heuristics, programmatic checks, and more traditional high-precision classifiers, then ensemble them into something greater than the sum of its parts and much more robust than a pure LLM. The week before the taping she had added a bunch of summarization guardrails. If you are aggregating a summary across documents, you may want it accurate, concise, and not redundant. When asserts are not enough, Guardrails breaks the problem into smaller machine-learning tasks or smaller verifiable heuristics and tries to get an aggregate assessment.

Faithfulness, after the sponsor break, is a concrete recipe. Look at each sentence in the summary, similarity-match it against passages in the source, and get fine-grained attribution for where it came from. As a developer you experiment, decide your appetite for how far a sentence may drift, set a threshold, and filter out anything below it. Concision is the sibling check: inside the generated summary, make sure each sentence is diverse enough from the others, and drop the ones that are too similar. The move is to stop treating a text output as a single unit and independently verify the chunks.

The first-order benefit of LLMs is that they can do insane tasks. The second-order benefit, she argues, is that you can use the models as verification systems on themselves. You can put a large language model inside a validator, or, depending on your data and latency, a smaller ML model, and bring several verification strategies together. That is the philosophy for adding guarantees around problems that used to look too hard to check.

Labenz is wearing Waymark swag. The company makes videos: minimal information about a business in, a ready-to-watch video out, with a language model at the core writing the script and directing visual assets. A year and a half earlier they had to fine-tune GPT-3 just to respect the shape of the task. GPT-4 can now do it zero-shot. GPT-3.5 Turbo is not reliable in the same way, and it is something like 20 times cheaper. If Guardrails can keep a less reliable model on format, the value drivers he can see are cost and latency. He asks what the community is actually buying.

Cost, latency, and the ability to use open-source or cheaper models at the same reliability are one aspect, Rajpal says. The more interesting question is what you are truly using the model for. Notion AI Assist, which she enjoys, helps you shape an outline; there, the creativity of a large language model is a desirable trait. The other cluster of use cases leans on the world model inside the LLM and treats it as a software abstraction for general-purpose reasoning. An AI receptionist for a small business — answering calls, figuring out scheduling, never asking a customer for private information — is a powerful capability in which creativity is not a good thing. You want it to stick to a workflow. Guardrails is most useful in that second world, where reliability is the point.

Over the weekend a community contributor had shipped Auto PR, a GitHub Action that takes an issue and automatically creates a pull request against the codebase. The constraints are strict: the files must exist, the diffs must be valid for those files. Those are hard to enforce without a validation framework on top. Even the creative use cases, she adds, still have implicit constraints people currently enforce by prompt iteration: no profanity in generated scripts, no mention of competitors in video copy. Encoding those and running a validator only when needed is nicer than doing it by hand.

She is careful not to oversell. Getting output in front of an end user, taking qualitative feedback, and going back to the drawing board is a valid way to work, and in some domains that human write-off is essential. Guardrails’ bet is that some of that feedback can be done by a mix of traditional ML, heuristics, and LLMs in the loop. If you can codify the qualitative criteria, you can generate specific error messages that help the model self-correct. Under the hood the library packages relevant context, builds a new prompt, re-asks only the parts that were wrong, and merges the new response with the old one. That self-heal loop did not exist before these models. It does not work for “the text must be funny,” a scoring function she cannot imagine, or for high-stakes domains where a human confirmation is still required before another iteration.

Everyone has agents on the brain, Labenz says. Dig in, and you usually find a multi-agent system that is the same language model playing different roles: a simulated town where GPT-4 is every resident, or a research agent with a planner, a coder, and a retriever, each with its own prompt, scaffolded together. Those systems fail a lot because there is some probability of being wrong at every link, and in the naive implementation you are only as strong as the weakest one. What he is learning is that the Guardrails paradigm is connective tissue between roles, and he should probably accelerate how quickly he thinks these agents will start to work.

Rajpal draws a contrast. Guardrails’ main problem is adding constraints so an output works for your use case, which gives developers a lot of agency over what correctness means. Agent frameworks today configure goals and tasks autonomously, by the language model itself. You typically do not get a fine-grained place to enter the framework and add guarantees, because you are not the person configuring the agents. If you actually want to employ a bunch of them to do research, you want a set of allowable things they can look at, configured dynamically based on where they are operating. Constraints and correctness specs, she says, are essential. It is the only way to assess what the models are doing at each step, and the key unsolved problem is how to do that dynamically when you did not set the goals. That problem needs to be solved before agents are truly employable outside of demos.

Waymark, with its finite scope, is probably not a great agent fit. Labenz splits interaction into a real-time co-pilot that shepherds you and a delegated mode where you put AI in the driver’s seat and look at the work later. He is advising Athena, an executive-assistant company, where humans still execute a lot of web tasks. Understanding a client request, parsing it, and asking follow-up questions already looks strong. Hitting the right buttons, checking out, logging in — two-factor authentication still deters AI — is where the systems fall down. Rajpal’s experience matches: you need grounding, and validation at each decision and action. Booking a flight given a schedule, a budget, and a destination is a bundle of constraints that have to be grounded in a calendar and in what flights exist.

Having worked in self-driving, she also names a trust deficit: even if the agents were perfect, people would still have a limited appetite for delegation, so you need a verification system that keeps control and oversight at each step. Labenz takes the other side of the bet. He thinks she is saving people from themselves. Users will be quick to say it seems to work, and what is the worst that could happen. He had just recorded Harvard Medical School professor Zak Kohane, author of *The AI Revolution in Medicine*, whose recurrent finding is that GPT-4 is better at evaluating text than at generating it. Rajpal’s validation system is trust-but-verify: generate, then evaluate in a separate step, and ensemble diverse evaluators. Tesla Full Self-Driving, which a neighbor demonstrated for Labenz, already engineers that balance with a three-step reminder — visual, then sound, then a pull-over warning, and eventually ejection from the program. The interface that lasts, she says, calls humans when they are most needed, because a constant “pay attention” stops working.

Labenz wonders whether a year from now a GPT-4-class central process could just use a Guardrails spec as it delegates work to little agents. Rajpal’s core belief is that humans stay essential in the loop, because correctness means very different things to different people. Profanity filtering is the rare consensus case. She has also talked to teams building chatbots where authenticity matters, and if the bot is in the likeness of someone who uses profanity, filtering it out actually hurts the experience. Those constraints are hard to set globally. Offloading the whole problem to the model or to a provider is going to be hard. The developer experience she can imagine is configuration-first: even if the underlying model stays the same, how it is validated, corrected, and post-processed is something you pass in every time. The prompt is not sufficient, and neither is choosing a model and a temperature.

Security is the other driver coming out of the mist. GPT-4 already stays on format most of the time, Labenz says, but prompt injection, Bing’s willingness to be manipulated, and whatever SEO people will do to AI search are barely started. There is model risk too. His current strategy is to use GPT-4 for everything, get quality acceptable, then take out cost and latency later. People will hear that some Alpaca or bootleg LLaMA fine-tune is just as good and drop it in. A huge misconception is that OpenAI’s or Anthropic’s safety posture is an inherent property of language models. They worked hard for it. You cannot take for granted that a fine-tune inherits the SLA.

Rajpal does not think this is a problem machine learning will solve by itself. OpenAI’s gains ride responsible scaling laws and emergent properties, but these are fundamentally stochastic systems. There is a long tail of exciting and weird ways people will use them, and you cannot have data points for all of them. You cannot add watertight guarantees from inside the model. Security, therefore, needs more determinism around it. Her decomposition is a sandwich: input validation and output validation around the LLM API call, multiple layers so the model is not behaving in ways you do not want. Known prompt-injection patterns can be gatekept on the way in, limiting users to a supported set of queries. Unwanted behaviors can be secondary-checked on the way out.

She has talked with teams already in production, and the design is common. There is a reason we do not have end-to-end machine-learning systems. You have a machine-learning subcomponent, then a checkpoint where a human or a more deterministic component validates the output, then another downstream piece. Defense in depth, Labenz agrees, is going to become really important.

The next version of the problem is models talking to other models in vector form. Labenz points to BLIP-2, whose authors had been on the show: a frozen vision model, a frozen language model, and a quickly trained connector that translates an image encoding into something unspeakable in the language embedding space, so you can have a dialogue about the image. Rajpal loves the example because it is literally her master’s research — joint embeddings in the fashion domain, projecting images and associated text into one space so language can retrieve similar images. He sees simple linear maps bridging latent spaces as a huge trend. You will not, long term, squash a medical scan or a bird call down into the sentence “a sound of birds is heard.” The worry is obvious: the possibility space is vast, untestable on the input side. His advice is blunt. Start validating your outputs. It is only going to get more important, and a system that behaves predictably today may expose its vulnerabilities as the world underneath it changes.

She takes the other tone on extra sensors. More sensory input leads to better performance, and it is another form of grounding: if two projections of the same scene should agree, you can use that as self-verification. At Drive.ai, lidar alone versus lidar plus cameras produced a better understanding of the car’s state and better decisions. Extra vectors in the same space, she says, let you see where things break down and add checks across dimensions. What she is describing, Labenz translates, is a validation step on the coherence of semantic interpretations from different signals.

He throws other safety paradigms at her, starting with OpenAI’s moderation endpoint: bump an output against a finite list of problematic content types, get a flag, and do what you will. She thinks it is great and very finite. Guardrails’ move is to expand that pattern to any use case that is programmatically or ML-verifiable, flag the problematic pieces, and let you configure how invalid outputs are dealt with. Profanity filtering is one validator among many in the library. Take the same pattern to generated code that will not execute, summaries that are unfaithful, or structured data with a wrong field.

On failure, she will not prescribe pain tolerance. If a failed output is of no use, you want it corrected or filtered; you cannot live in the in-between. Re-asking is the one she sees used a lot on high-stakes work: from the user’s perspective it is a single API call, even if the backend makes several, and you either get a corrected output or a hard “I cannot handle this.” Filtering is the granular cousin, dropping specific sentences from a summary or stripping profanity. The default setting is a no-op: still run every validation, return the output as if Guardrails were not there, and log everything that went wrong so you can decide whether to iterate on the model or the prompt. You can also raise an exception, or deterministically fix the output when a deterministic fix exists.

OpenAI’s Evals library, shipped with GPT-4, looks to her like an offline benchmarking suite for tasks people care about, not a runtime aid. What she likes, more as product than as technique, is that it crowdsources how people want to use the models and how those uses go wrong, then folds that into training and evaluation. Other folks mine it for interesting prompts. Anthropic’s Constitutional AI is interesting as a training protocol, but it lands on her earlier point: training alone is not sufficient. Runtime verification is the exciting problem, and configurability matters. You should be able to enforce what correctness means to you without a globally understood standard, and without going off to train a new model every time the definition changes.

Labenz notes that “deterministic” is doing a lot of work, because an LLM critique is still stochastic. She qualifies. One path is to take an output, build a new prompt, and ask GPT-4 whether it is correct given criteria — extra security, not deterministic at all. Other validators are rule-based. Others are smaller high-precision models trained on your own data, where you control the seed and the randomness even if the model is still a model. When she says deterministic she means an ensemble of those techniques, all over the map, stacked.

Overhead — tokens, complexity, latency, and the product tension of streaming — is the developer objection. Microsoft’s Bing, he notes, streams tokens and then retracts them if they go off the rails, which is a wild production choice for one of the world’s largest companies. Rajpal was surprised, outside of chatbots, how comfortable people are with extra latency if the final output is high quality. Re-asking is designed as one aggregated request, not piecemeal. Token overhead can even go the other way: structured prompting in a domain-specific language is sometimes cheaper than putting every constraint into English words. Complexity drops because you stop iterating prompts by hand. The entry point is a RAIL spec — Reliable AI Markup Language — where the output schema is totally separate from the prompt. The prompt holds the high-level task description; formatting and constraints live in markup you can write if you can write XML. Guardrails’ job is to compile that spec into a prompt. When a provider silently updates a model version and behavior shifts, the developer keeps the same RAIL spec and the library compiles a different prompt for whatever is most effective. A community contributor added instruction-tag experiments so GPT-3.5 just works, without the “hope this was helpful” filler that used to break JSON.

On the market, Labenz worries that everyone building against OpenAI’s current behavior forces Anthropic, Cohere, Google, and Aleph Alpha to spend their energy impersonating OpenAI. Rajpal expects diversity. Demand is so loud that people who were not even in the space a few months ago are training and fine-tuning, and she has already heard “I like Guardrails, I like OpenAI, it is just too expensive — can you make this work with an open-source model?” Great models at different prices, latencies, and providers will proliferate. Standards and models are evolving at the same time, which for now incentivizes other labs not to regress on the functionality frameworks assume, and eventually a more level field with real specializations. In the quickfire, the AI tool she would not recode without is Copilot; she noticed herself slowing down on the days her dev environment was missing it. She would skip a Neuralink implant even if a million people had one, because a distracted thought in a presentation should not execute on the slide deck — she wants a filtration layer between thinking and doing. Her fear is job displacement as knowledge work is assisted. Her hope is a life where she does not do her own taxes or book her own flights.