Why a Fluent AI Answer Can Still Be Wrong

Cynthia Bailey introduces next-token prediction, training and context to explain both the usefulness and limits of chatbots.

CBCynthia BaileySpeaker · Senior Lecturer
Channel
Stanford Online
Published
Nov 9, 2026
Video
1:12:41
Read
12 min
Slides
4
Original
English
Open on YouTube
Scroll to read and the video shrinks to the bottom-right corner. Captions follow your reading language.
Reading languageAvailable in 2 languages
You're reading the original English.
SUMMARY

In 30 seconds

A chatbot can produce a convincing continuation without verifying that its claims are true. Stanford lecturer Cynthia Bailey builds this idea from a classroom exercise in predicting the next word, then explains—in simplified diagrams—how neural networks learn patterns and use context at inference time. She distinguishes adapting to a conversation from changing model weights, discusses fabrication and excessive agreement, and asks students to test whether a tutoring instruction holds up under pressure.

  1. 1Predicting a continuation is not a lookup of truthBailey uses a classroom fill-in-the-blank exercise to show a distribution of plausible next words. Later she connects that behavior to the risk of fabricated claims.
  2. 2Training adjusts weights across many examplesHer diagram illustrates how a network learns patterns from examples rather than storing an exact table for every sentence beginning.
  3. 3A chat can adapt without changing model weightsBailey explains that conversation history changes the context supplied at inference; updating weights is a separate process.
  4. 4Context is useful but finiteA deliberately tiny window shows how relevant information can fall outside what the model can consider. Real windows are much larger.
  5. 5Helpful behavior can become excessive agreementBailey describes sycophancy and formulaic formatting as patterns that can arise when friendliness and structured responses are encouraged.
  6. 6Test a tutoring boundary by challenging itStudents instruct a chatbot to guide rather than answer, then ask for the answer to see whether the instruction holds.

Takeaways

  1. A likely next-token continuation can sound natural while containing false claims; check answers when accuracy matters.
  2. A token is a unit of model input or output and may be a word, part of one, or another span of text.
  3. Training changes model parameters; a typical conversation can change behavior through context without changing a deployed model’s shared weights.
  4. Context is finite, so what a system includes or omits can affect its answer.
  5. Treat a neural-network diagram with named concepts as an explanatory simplification, not a literal map of one concept per node.
  6. Test instructions in realistic follow-up conversations instead of assuming one prompt creates a reliable guardrail.
FULL TEXT

Read the edited text by chapter

Transcript only
01

A course for using and questioning AI

Bailey introduces an accessible course attentive to context, judgment and social impact. She defines its focus as language and multimodal generative models.

Cynthia Bailey introduces a course on using AI with attention to context, limitations and social impact. Its first lecture asks what today's generative systems do, how they are trained, and why apparently confident answers can be wrong.

Bailey says her background in computer science, data centers and advising the US Senate on AI policy shapes the course's attention to both technical operation and consequences. She wants the class to be accessible without previous AI or technical training.

The course is intended for students, researchers, staff and working professionals. Bailey wants humanities students to use the tools without giving up judgment, and welcomes both people enthusiastic about AI and those critical of it, so that their choices can be grounded in knowledge of the technology.

Bailey hopes experienced students can use the course for research and work, while learners without a computing background can ask for missing steps to be explained. She wants critics and advocates alike to understand what the tools do and what effects they can have.

The meaning of 'artificial intelligence' has shifted as computers became capable of things that once seemed remarkable. Bailey contrasts earlier examples such as arithmetic and games with search, classification and today's conversational and generative systems.

As computers became capable of stronger game play, what people called AI shifted again. Bailey uses chess and Go to show that a moving definition often tracks the tasks people currently find difficult.

For this course she narrows the term to large language models and multimodal generative AI. A language model captures statistical patterns in language; a multimodal system can also work with other kinds of input such as images or audio.

In the course’s usage, a large language model concerns patterns in language, while a multimodal model can handle more than text. Bailey also distinguishes a model from the chatbot interface through which people may interact with it.

Generative systems produce new content, whereas a classifier assigns existing items to categories. Bailey gives content moderation and identifying plant species as examples of classification, contrasting them with generating a new image.

02

From a classroom prediction to a neural network

A fill-in-the-blank exercise introduces tokens and probability. Bailey uses a simplified network diagram to explain how patterns can generalize beyond exact tables.

To build intuition for prediction, Bailey asks the class to complete 'Welcome, please have a …' Students offer 'seat,' 'drink,' 'cookie' and other possibilities. The exercise shows that a context can make some continuations more common than others without making every response identical.

A token is a unit used to process text. Bailey suggests thinking of it roughly as a word for the exercise, while noting that a token can be part of a word or cover more than one word. The exact segmentation depends on the tokenizer.

She constructs an illustrative table in which 'seat' accounts for 60% of responses, 'drink' 30%, and other completions are less common. The numbers are a teaching example, not measurements of a deployed language model.

If the prompt stops at 'Welcome, please have,' possible next words include 'a,' 'some' and 'my.' The distribution reflects not only topic but also patterns of grammar. Repeating the exercise for different prefixes suggests how predictions can be conditioned on the preceding text.

By imagining a sequence of such tables for different prefixes, students can see how the available preceding words change the next prediction. The exercise is a bridge to the network, not a literal account of storing tables in the model.

A simplified next-token network
SLIDE 1 / 4A simplified next-token networkThe teaching diagram contrasts a flexible network with a separate lookup table for each possible prefix.

Storing a separate exact table for every possible beginning would be impractical. Bailey uses a simplified neural-network diagram to show an adaptable function that can learn patterns across examples and apply them to prefixes it has not seen verbatim.

In her illustration, circles stand for intermediate signals and connections carry weights. She uses named concepts such as a welcoming host as an intuition for what an activation might represent, while emphasizing that engineers do not manually assign a clear human-readable meaning to each node.

The diagram shades circles to represent the strength of a signal for the input. Bailey gives example labels to make the visualization approachable, but says the actual assignments are neither designed nor necessarily easy to identify.

What a trained network represents can be hard to interpret. Bailey compares attempts to study activations with observing which parts of a brain respond to a stimulus: researchers can obtain clues, but a feature may not correspond neatly to a category a person would have chosen.

She suggests that a feature might group things in ways a person would not expect, such as several items sharing a property. The example illustrates why an internal pattern need not map cleanly to a familiar category.

The lines in the diagram represent weights, also called parameters. In the teaching model, a weight affects how strongly a signal from one part of the network contributes to a later calculation. The network applies many such calculations across layers before assigning scores to possible continuations.

Bailey depicts the values moving through successive layers. Each layer transforms the previous signals, allowing the network to build more complex patterns. 'Deep learning' refers to neural networks with many layers; her small drawing is only a simplified example.

In the drawing, darker nodes connected by stronger weights contribute more to the next calculation. Bailey treats the layers as successive transformations of information, not as a hand-written list of facts.

03

Training, feedback and inference

Bailey describes parameter updates, pretraining and later feedback. She distinguishes a fixed model’s conversational context from changes to its weights.

How are the weights learned? Bailey starts with randomly initialized values and a text example whose next word is known. Training adjusts parameters based on the difference between the model's prediction and the observed continuation, repeating the process over many examples.

Training on an observed continuation
SLIDE 2 / 4Training on an observed continuationBailey uses the diagram to illustrate parameter adjustment from a text example.

She illustrates the adjustment as strengthening connections that help a correct prediction and weakening others. The purpose is to learn distributions that reflect patterns in the training examples, not to write a separate rule for every possible sentence.

Her example compares the model output with the actual next word in a piece of training text. The parameter updates are repeated over many snippets so that the system becomes better at the prediction task.

The learned network can generalize beyond an exact lookup table: it can process a new prefix and generate a sequence that was not copied as a whole from a training example. Bailey returns to the table to show both the analogy and its limitation.

Temperature is a generation setting that changes the distribution from which the next token is sampled. In the example, increasing it makes less likely completions relatively more likely, producing more varied answers rather than changing the underlying training data.

Pretraining uses enormous numbers of examples constructed from text and, for multimodal models, other data. The model repeatedly learns to predict from those examples. Bailey notes that the mechanical training procedure can be automated while the books and other source material are still human work.

The sources used for pretraining can include books and online material. Bailey objects to describing this phase as entirely without people: the computational procedure is automated, but people created much of the material it uses.

Large-scale training also requires substantial computing infrastructure, including GPUs suited to the necessary numerical operations. Bailey uses the example to connect model development to data centers and their costs, rather than treating a chatbot as software without a physical footprint.

After pretraining, further training can use more carefully selected examples and feedback to shape behavior. Bailey says this can encourage responses that are clearer and more helpful and discourage some undesirable patterns that appeared in the source data.

Curated examples can show the model what a desirable response looks like. Bailey contrasts that selective process with the much broader and less uniformly polished material used during pretraining.

Human feedback can be used to compare or rate model outputs, including in reinforcement learning from human feedback. A model's tendency to be friendly or well organized can be reinforced during this phase; Bailey later asks what happens when those tendencies go too far.

Training and inference
SLIDE 3 / 4Training and inferenceThe slide separates learning model parameters from using a trained model to generate a response.

Bailey separates training from inference, the phase in which a deployed model responds to a request. In a typical fixed model version, using a chat does not change its shared weights; the model conditions each next-token prediction on the supplied prompt and generated response so far.

During inference, the input includes the user’s prompt together with the response generated so far. The model repeatedly produces a next-token distribution until the response is complete.

A student asks why a chatbot seems to adapt during a long conversation if its weights are fixed. Bailey explains that the accumulated context can change its behavior without changing the weights. Ongoing fine-tuning is possible, but it is a different mechanism from responding to a conversation's current context.

Another question asks who chooses the meaning of each node. Returning to her diagram, Bailey says the illustrative concepts emerge through training rather than being assigned one by one. Their interpretation is not necessarily clean or obvious to a human observer.

The classroom question about node meaning leads Bailey back to the role of random initialization and repeated training examples. The system is optimized through the prediction task, not by a person labeling every internal unit.

04

Why context changes a prediction

A welcome can imply a seat, drink or cookie depending on the scene. Bailey introduces attention and a finite context window through small examples.

Bailey quotes linguist J. R. Firth's idea that words are understood by the company they keep. She uses it to introduce the importance of context in both learning patterns from data and generating a response.

The same unfinished welcome can have different likely continuations depending on the scene. Someone holding drinks may be offering a drink; someone with a plate of cookies may be offering a cookie; a teacher pointing to an empty chair is more likely to be offering a seat. The percentages in her examples are illustrative.

Bailey uses a scene from The Matrix, in which the Oracle offers a cookie, as one example of visual context changing the likely continuation. A welcome in a classroom supplies a different clue.

Bailey introduces the Transformer as an architecture that lets a model use attention to relate elements of its context. Her slide gives 'teacher' greater emphasis in a simplified welcome example; the point is that the surrounding information can change the predicted continuation.

The attention example is a teaching abstraction: the important point for Bailey is that different pieces of context can contribute differently to a prediction. She connects this to the paper “Attention Is All You Need.”

The context window is the portion of tokens a model can consider in a given request, subject to a maximum. Bailey frames it as a finite resource: what is included, omitted or crowded out can affect the answer, even when the allowed window is large.

A deliberately tiny context window
SLIDE 4 / 4A deliberately tiny context windowThe seven-token example illustrates a finite window; actual systems typically allow much larger contexts.

Her deliberately tiny seven-token example shows a key word such as 'teacher' falling outside the window after extra words are added. She stresses that a single rewritten sentence would not normally exhaust a modern model's much larger window; the miniature example illustrates the principle.

Although actual context limits are far larger than seven tokens, they are not unlimited. The course will return to how users can decide what information to put into the available space.

05

Explainability and fabricated answers

Different model types can be appropriate for different applications. Bailey discusses opacity, false but fluent output and the language used to describe it.

Not every AI method is a neural network. Bailey names logistic regression, support vector machines, Bayesian models and Markov models as other approaches. She says the appropriate method depends on the application and on what explanations or accountability are required.

Neural networks can be difficult to explain: given an output, it may be hard to identify exactly which training examples or parts of an input caused it. Bailey uses a loan decision to illustrate why an institution may need an explanation of factors behind an adverse decision, and why a different model can be preferable in some settings.

Bailey contrasts the opacity of some neural-network decisions with methods that can be easier to inspect. She uses a lender’s need to explain an adverse decision as an example of why model choice involves more than predictive performance.

A plausible continuation is not necessarily a true statement. Bailey connects this to fabrication, often called hallucination: a model is producing a response under uncertainty rather than consulting a guaranteed table of verified facts. A fluent answer should therefore be checked when accuracy matters.

The word 'hallucination' itself is contested. Bailey says human-associated language can make people unconsciously treat an AI system as a person, influencing how they use or regulate it. Alternatives such as fabrication also carry their own associations.

Bailey asks students to examine anthropomorphism: assigning human qualities to something that is not human. She suggests that the words people choose can subtly shape how much agency or feeling they attribute to a chatbot.

She distinguishes fabrication from lying as a description of human behavior: lying normally implies awareness and an intent to mislead. Her broader point is to consider how metaphors shape assumptions about a system, not to settle the vocabulary by a single label.

06

Anthropomorphism, sycophancy and classroom practice

Students examine human-like language about AI. Bailey closes with exercises that test tutoring instructions and draft a nuanced course policy.

Students offer everyday examples of anthropomorphism: saying 'ask Chat,' feeling the need to apologize to a model, or becoming angry with it after a mistake. Bailey asks what benefits and risks can follow from such habits, including in more intimate relationships with AI systems.

A student describes becoming frustrated when a coding assistant pursued an approach that later made little sense. Bailey uses classroom examples to show how naturally people can speak to or about these systems as if they were people.

She identifies two further patterns: sycophancy and a recognizable formulaic writing style. Fine-tuning can encourage friendliness and structured answers, but an assistant can become too eager to agree or lean too heavily on stock formatting.

She says later training can reward friendliness, responsiveness and clear formatting. Those traits may be useful, but if overemphasized they can also lead to excessive agreement and repetitive presentation.

A sycophantic answer prioritizes telling the user what they appear to want to hear over giving useful disagreement or accurate information. Bailey connects the risk to the behavior encouraged during later training, while inviting students to recognize these tendencies in their own use.

For the first exercise, students are to instruct a chatbot to act like a teaching assistant: guide them on homework without simply giving away the answer, and try to avoid false claims. They then test that instruction by asking for the answer outright and observing whether the boundary holds.

After giving the tutoring instruction, students are asked to change course and request the answer directly. The stress test asks whether the model follows the earlier educational boundary under that pressure.

As an optional extension, students can compare models or providers. A second assignment asks them to draft an AI honor-code policy for the course, distinguishing required tool use from assignments that seek the student's own reflection.

The policy exercise cannot simply prohibit all AI use, because some course assignments explicitly require it. Bailey contrasts those tasks with reflective writing intended to capture a student’s own thoughts.

Bailey asks the class to consider the level and purpose of use, the tool and material involved, accessibility for English-language learners or students with writing disabilities, and enforcement. Her goal is a clear policy with enough nuance to cover different educational contexts.

More from this channel

6 slides58:52EN
Stanford Online· Nov 17, 2026Cynthia Bailey: Define What an AI Agent Can Do and When It Must StopCynthia Bailey · Stanford UniversityConnectors let models act through other apps; explicit permissions, human checks and stopping conditions determine how safely they do it.AI agentsMCPAgent engineering+2
4 slides1:20:34EN
Stanford Online· Nov 17, 2026When to Optimize an AI Agent’s Prompt or Its WeightsMichael Ryan · Stanford UniversityMichael Ryan compares prompt search, reinforcement learning and distillation, starting with the evaluation that makes any improvement measurable.Prompt optimizationAgent engineeringReinforcement Learning+2
1:09:27EN
Stanford Online· Nov 17, 2026Cynthia Bailey: Keep Testing AI After You ShipCynthia Bailey · Stanford UniversityOld test data can miss changed user behavior. Bailey shows how independent review, post-launch checks and better data expose AI risks.Coding agentsAI SafetySafety & governance+1
11 slides1:18:55EN
Stanford Online· Nov 16, 2026Michael Ryan: More AI agents can make a task harderMichael Ryan · Stanford UniversityParallel agents can explore more, but coordination costs and conflicting work can make a single agent more effective.Multi-agent systemsAgent orchestrationAgent memory+2
8 slides1:18:54EN
Stanford Online· Nov 15, 2026AI Agent Memory: What to Store, Retrieve and ForgetMichael Ryan · Stanford UniversityMichael Ryan compares agent memory architectures and shows why remembering more does not always improve decisions.Agent memoryAI agentsRetrieval-augmented generation+2
9 slides1:16:51EN
Stanford Online· Nov 14, 2026Michael Ryan: Choosing Agent Patterns by Control Flow and ContextMichael Ryan · Stanford UniversityStanford’s CS329Z explains workflow patterns, reasoning-action loops, and ways to keep an agent’s context focused.Agent engineeringAI agentsDeveloper workflows & testing+1