En 30 secondes
An AI tool can pass extensive tests and still fail once people know it is in use. Stanford lecturer Cynthia Bailey describes a grading assistant tested on years of assignments that may encounter prompt-injection instructions absent from that history. She recommends a detailed specification and a fresh reviewer for AI-written code, then ongoing monitoring after release. Her firing example shows why observed outcomes can hide mistakes when decisions remove the very people whose results would reveal them. She also explains how to map and manage risks and question whether convenient datasets represent ground truth.
- 1A fresh reviewer can check AI-written code against the specificationBailey suggests giving a separate coding agent the scope and output, asking for a skeptical review, and including tests in the original plan.
- 2Historical grading data can miss prompt injectionAn AI grading assistant performed well on old assignments, but students aware of the tool could add instructions that never appeared in those records.
- 3Firing every flagged employee hides the model’s mistakesThe company sees outcomes only for people it retained. Bailey describes retaining a random sample of those flagged to learn about the missing counterfactual.
- 4Risk management needs action as well as measurementThe NIST framework connects mapping and measuring risks to managing them, with organizational governance at its center.
- 5An accurate display of records can still misrepresent realityPolice records reflect what was reported and detected, not all crime. Bailey uses this example to show why a convenient dataset is not automatically ground truth.
- 6Users can test the behavior of deployed algorithmsA Twitter image-cropping example prompted people to swap image positions and try other pairs, revealing behavior they wanted to evaluate.
À retenir
- Give a fresh coding agent the specification and output to review, and include tests in the plan from the start.
- Recheck deployed systems when users, conditions or model behavior change; historical accuracy is not a guarantee.
- Look for missing counterfactual outcomes when AI decisions change who remains observable.
- Treat available data as a measurement with limitations, not automatically as ground truth.
- Plan for data handling throughout a product’s life, including decommissioning.
Lire le texte remanié, chapitre par chapitre
Using AI to build a simpler tool
Bailey introduces vibe coding and explains when a reusable software tool can replace repeated model calls.
This week’s tutorial uses coding agents such as Claude Code and Codex. I want to introduce a process for making software with them, then turn to how to stay in control of AI through risk management and evaluation.
Vibe coding means using ordinary language to direct software creation. Early experiments were often exciting but produced buggy code that was not ready for production. More deliberate approaches are emerging; some people call the professional version agentic engineering.
The process we discussed with an elephant and a goldfish is one influential approach to professional work. I use vibe coding broadly enough to include both playful experiments and more serious engineering.
One useful mental model is to have AI write itself out of a recurring task. Instead of asking a large model every time which bills a member of Congress has introduced, use AI to help create a dashboard that retrieves the information.
Then the ordinary software tool can do that repeated retrieval without a large model each time. This can serve the task while also reducing computing and token costs. If a calculator or a search function already does what you need, you may not need an LLM at all.
If no simpler tool exists, coding with AI can help you make one for your specific need. The answer to ‘is there already a simpler tool?’ can change from no to yes once you create it.
Prepare, implement and review with separate agents
A detailed specification, an implementation agent and an independent reviewer keep a human involved.
A heavy process is appropriate when software accuracy affects important outcomes, such as the reliability of research. For this week’s tutorial, however, I also encourage you to experiment and see how simply you can describe a small website.
The three steps are to prepare, generate and review. Preparation means using the elephant-and-goldfish process to develop a detailed scope of work, including what the tool should do and how its quality will be assessed.
Give that specification to a fresh coding agent. This ‘goldfish’ has not participated in the earlier discussion, but it is equipped for software implementation. A detailed spec should reduce the number of questions it has to ask.
When an agent asks you to choose a database or another unfamiliar component, do not assume you must accept its recommendation. Ask it to explain the background and alternatives so you can make a meaningful decision.
Review is still possible if you cannot read code yourself. Start another fresh coding agent, give it the specification and the resulting code, and ask it to conduct a skeptical professional review.
To push the reviewer away from merely praising the result, frame the code as someone else’s work that may not follow the specification. The aim is to get a thorough review while keeping the human involved.
A student asks about testing. Unit tests and other checks belong in the plan from the beginning: implementation agents can run them as they work, and the reviewer can run a final pass.
Another student asks how a fresh agent can understand constraints imposed by a larger codebase. Put those constraints in the specification and cite the relevant documents or code files, including interfaces the new work must use.
Why testing must continue after launch
Changing conditions, user behavior and model updates can create failures that old tests miss.
Building and checking a tool is not the end of its life cycle. Once a tool is deployed and people begin using it, new problems can arise even after careful design and extensive testing.
One cause is distribution shift: conditions in the world can change relative to the data on which the tool was tested. A fixed model may then produce new failures even though those failures were not apparent before deployment.
Users can also change their behavior because they know they are interacting with AI. They may act differently for benign reasons, or they may deliberately include instructions intended to manipulate the system.
In another class, unexpectedly high enrollment led our staff to build an AI tool to help grade programming assignments. We compared its grades on many years of past submissions with human grades and saw strong results.
But historical submissions gave students no reason to write a note aimed at an AI grader, such as an instruction to award full marks. Once students know about the tool, that kind of prompt injection may appear despite thorough testing on old work.
A deployed system can also change through its context or, for systems designed to learn after release, through updates to model weights. Those changes can produce behavior not observed in pre-deployment tests.
I note that some military AI experts recommend against deploying weapon systems that continue learning in the field. Their concern is that changing behavior could create harmful effects outside the intended scope.
The missing outcomes in an AI firing decision
A hypothetical hiring example illustrates survivorship bias and the need for counterfactual data.
Consider a company that hires people for 12 months and wants to predict, after 90 days, who will perform poorly. It trains an AI on part of its historical employee data and tests predictions against another held-out part.
Suppose the test appears flawless. The company deploys the system, fires everyone it flags after three months and later observes that the retained employees perform well. Does that prove the tool was right about everyone?
No: the company lacks outcomes for the employees it fired. This is survivorship bias. Observing good results among those retained does not show that all of those dismissed would have performed badly.
The missing information is the counterfactual: what would have happened if the flagged employees had stayed? One experimental design would randomly retain 25% of those flagged and observe their outcomes, rather than firing all of them.
Such a comparison can provide evidence about the wider flagged group, although the decision itself has consequences for people. The general measurement problem arises whenever an AI-informed decision changes which outcomes can later be observed.
Map, measure, manage and govern AI risk
The NIST framework offers contextual guidance across an organization and a product’s life cycle.
The National Institute of Standards and Technology released its AI Risk Management Framework in 2023. Unlike an exact physical standard, Bailey describes it as guidance and questions organizations can use to build responsible practices.
Map means identifying the intended use, data sources, deployment setting, algorithm choices and possible harms early. Asking ‘what is the worst that could happen?’ is one way to begin mapping the risks.
Measure asks an organization to develop ways to assess risks and performance in the actual context of use. There is no single precise unit of AI risk, so measurement often requires new work and judgment.
Manage means using the results of mapping and measurement to do something about the risks. Listing worst cases is not enough without deciding how to mitigate them.
Governance sits at the center of the framework. A policy written by consultants will not help much if nobody reads or follows it; Bailey emphasizes the long-term work of making safety part of organizational culture.
The framework asks teams to attend to the whole product life cycle. Consider data sources and potential consequences during planning and design, not for the first time the morning after launch.
The post-deployment phase matters, but so does decommissioning. A retired product may leave sensitive data in a database or on a drive, so teams need a plan for what happens to it when the product ends.
I once moved a small startup into an office vacated by a failed business. The old occupant had left files containing sensitive customer financial information, which we had shredded. That is an example of failing to plan for the end of a service.
Apply data rules and empathy
Stanford’s data classifications and its guidance about others’ information inform responsible tool use.
Stanford’s own AI policies matter to students who may handle other people’s information as teaching assistants, researchers or club officers. Responsible use includes following the institution’s rules for the data involved.
Stanford classifies data by risk and maintains information about which AI tools and connectors are approved for each category. Some arrangements include contractual protections for submitted data; approval for one tool does not automatically extend to every connector.
A principle in Stanford’s guidance is to avoid putting information about others into generative AI that you would not want them to put in about you. Its open-endedness asks people to pause and exercise empathy, not merely follow a checklist.
I extend that question beyond the university policy: would you want cameras following your family, a data center beside your home, or your creative work used for training without credit or compensation? These questions can prompt discussion about choices made for others.
Other examples include personalized pricing, pay decisions informed by private financial information, and altering someone’s image for promotional use. Bailey asks students to consider the effects on the people whose data or images are involved.
Evaluate the model, not just product growth
Bailey contrasts model evaluations with business metrics and describes model-assisted judging.
Evaluation of a model differs from product metrics. Product metrics ask whether users are growing or engaged; model evaluations ask questions such as whether responses are biased, hallucinate or use tools correctly.
A model can be evaluated before deployment and should also be checked afterward. Business metrics depend on the product having users and therefore become available in use.
To estimate how often a model hallucinates, you need a way to judge each of many responses. Qualities such as tone and bias can be nuanced and difficult to reduce to a simple keyword or yes-or-no rule.
One way to make these judgments at scale is to use a second model as a judge of the first. The fresh agent reviewing another agent’s code is an example of this pattern, though the evaluation still needs careful design.
Many evaluations must be designed for the specific tool, but public benchmarks and leaderboards can help compare models. Bailey notes that some policymakers hope the present evaluation ecosystem could eventually support more formal auditing.
Convenient data is not always ground truth
Police-record maps and Twitter’s image-cropping tests illustrate sampling and evaluation challenges.
An evaluation also needs some account of the correct answer, or ground truth. That is harder than it sounds: a convenient dataset may contain sampling bias or omit important outcomes, as in the fired-employee example.
Bailey draws on sociologist Ruha Benjamin’s discussion of maps built from police records. A map can accurately display the records while still failing to represent all crime, because many offenses never enter those records.
For instance, if police set up driving-under-the-influence checkpoints more often in one neighborhood, recorded incidents may reflect that deployment as well as underlying behavior. The dataset is not the same as the thing the map appears to measure.
A parody map of white-collar crime illustrates another choice: categories excluded from a dataset can disappear from the picture even when they cause harm. Deciding which categories to include remains a design responsibility.
A dataset of convenience is available because it was collected for another purpose or produced as a byproduct of another process. It may be cheaper to reuse than to collect the ideal data, but its omissions can travel into the new application.
In a 2020 example, Twitter used an algorithm to choose a preview crop from tall images. People tested an image containing Mitch McConnell and Barack Obama and saw the preview select McConnell even after swapping their positions.
Other users then tested more image pairs, including performers and cartoons. Bailey treats the community’s experiments as an example of public evaluation and argues that more testing before launch would have been valuable.
What questions would you like answered about the models you use? Notice a behavior that frustrates or concerns you, then consider how to turn it into a more systematic experiment.
Sur la même chaîne
58:52EN
1:18:55EN
1:18:54EN
1:16:51EN
1:17:14EN
1:40:43EN