How language models learn from text and improve by trial and error

Jeremy Berman sets out to explain how a file of numbers and a comparatively small program can produce an intelligent system. He starts with neural networks and training before moving to language-model pretraining, post-training, reasoning and the possibility of new discoveries.

Matthew Berman notes that Jeremy is a self-taught researcher. Jeremy recommends following one’s curiosity into the parts of the field that become interesting; each component can be studied much more deeply than this overview allows.

Consider a list of New York apartments with location, size, bathroom count and sale price. A network learns a function mapping those inputs to a price. A person could code rules for a few examples, but hand-written rules become unwieldy as the number of examples and variables grows.

In a small table, the correct prices are known. The computer’s challenge is to find a rule that generalizes across those examples rather than rely on one manually specified rule per apartment.

In a toy formula, each input is multiplied by a parameter—a number Jeremy calls a knob—and the results are combined. Random initial settings give poor price predictions. Adjusting the knobs changes the output, allowing it to move closer to the observed prices.

Matthew suggests thinking of the knobs as weights on the importance of the inputs. Jeremy agrees that this is useful for the simple example, while noting that more complicated relationships can involve interactions between inputs.

Jeremy points out that a setting can have different effects when inputs interact. The simple weighting analogy is useful, but it does not capture every relationship a multilayer network can learn.

A deeper model feeds the result of one layer into another. Each layer has its own numerical settings, so the output is produced by a sequence of calculations.

A network stacks functions inside functions, each with its own parameters. The training goal is to minimize loss, a measure of the difference between the network’s prediction and the result used as the target. In the apartment example the target is the actual sale price.

Matthew gives a concrete error: if a $2 million apartment is predicted at $1 million, the loss should push the training process toward a better estimate. Jeremy emphasizes that the same settings are assessed across examples.

The network diagram expresses the same idea at larger scale: numerical inputs pass through connections with weights and intermediate layers. A language model’s parameter count refers to numbers in the model; a program describes how computations use them.

Adjusting every parameter by hand would take too long. Jeremy introduces automatic differentiation tools such as PyTorch, which track computations and calculate how changes in each parameter affect the loss.

The software tracks how calculations are connected, which lets it work backward from the final error. Jeremy describes this as finding the effect a small change in each setting would have on the outcome.

The calculation uses derivatives: for each knob, the system estimates the local effect of changing it on the final error. Matthew asks for a plainer explanation, and Jeremy returns to the practical consequence: if raising a setting increases error, adjust it in the other direction.

Jeremy uses a hypothetical sensitivity of three: a one-unit increase in a parameter might increase loss by three. Knowing the direction tells the optimizer whether a small increase or decrease is locally useful.

After calculating the direction, training nudges the parameters rather than finding the entire answer in one move. The learning rate controls the size of each update. The same process is repeated across the parameters and examples.

The calculation is based on the chain rule from calculus rather than random guessing. The speakers leave the derivation aside and focus on the information the software returns.

In the demonstration, an automatic training loop adjusts the apartment-price parameters until error decreases. Jeremy says its simple linear formula has limits; additional layers and nonlinear transformations can represent more complicated relationships.

The demonstration reaches an average miss of roughly $0.07 million. Jeremy attributes the remaining error in the toy case to the limited form of the function, and describes extra layers and nonlinear functions as ways to represent richer patterns.

The technique for propagating error information back through the network is backpropagation. It depends on inputs and target outcomes: without an observed price, the example cannot calculate how far its prediction was from the value it is trying to learn.

Jeremy shows a small PyTorch training loop: calculate predictions, calculate loss, reset gradients, run the backward calculation and update parameters. The example prints results at intervals rather than after every one of its thousand steps.

In the sample code, the optimizer clears the previous gradients, the loss calculation runs backward, and a step updates the model. Jeremy cautions that the demonstration is not a production training setup.

The sample reports a mean squared error, which turns prediction differences into a loss for optimization. A larger value in the demonstration means the predictions are further from their targets.

When Matthew asks how the code knows which way to go, Jeremy points to the recorded computation and gradient. They indicate the local relationship between a parameter and the loss; the optimizer uses that information to nudge the parameter toward lower error.

A language model can use the same apartment data as a sequence of text rather than a fixed set of numerical inputs. It predicts the next token from preceding tokens; tokens can be parts of words or other pieces of text.

Matthew emphasizes the change in representation: the same neighborhood, size and bathroom count are now read as text in a sequence. The network predicts what follows rather than accepting the fields only as separate numerical inputs.

The model is looking for statistical patterns in the text it was trained on. With a table rendered as text, those patterns could include the price after a neighborhood, size and bathroom count, but a language model is a more general kind of neural network.

For a narrow task such as predicting a price, Jeremy says a specialized model can be preferable given finite compute and data. Language offers far more possible outputs and relationships, making it more resource-intensive, though the same broad principle of adjusting parameters applies.

The two approaches are related: both are neural networks, and both learn by adjusting parameters. Jeremy expects them to find similar patterns in an idealized limit, while stressing that practical resource constraints make the choice consequential.

Training on a much larger text collection gives a model patterns for completing many kinds of input. It does not guarantee truth: a base model is optimized to continue text resembling its data, not inherently to behave as a helpful assistant.

Instead of outputting a single price, a language model assigns probabilities across possible next tokens. Training adjusts parameters to increase the probability of the token in the training example. The loss measures how the predicted distribution differs from that target.

For one training example, imagine the observed next token currently receives a low predicted probability. The update nudges that token’s probability upward relative to alternatives; this is one example, repeated across very large quantities of data.

Jeremy demonstrates a small Qwen base model with about 500 million parameters. Given a starting prompt, it generates a continuation; some examples resemble code or Chinese text. These outputs show the model’s learned patterns, but they do not by themselves establish exactly what was in its training set.

The base model is an autocomplete system. Its generated snippets can reveal the types of patterns it learned, but Jeremy does not present a completion as proof that a particular document or quote was in the training data.

When prompted to continue “Matthew Berman said,” the base model produces a plausible statement Matthew does not recall saying. Jeremy explains that a continuation may sound consistent with its prompt without being accurate or a verbatim quotation.

The speakers discuss whether including more of Matthew’s actual words could make an imitation sound more like him. Jeremy cautions that resemblance still would not ensure the model repeats an accurate quotation.

The base model is not yet trained specifically to answer helpfully or stop at the right time. Jeremy illustrates how a prompt formatted as a user question followed by an assistant label can elicit an answer because similar patterns appeared in its training.

A manually supplied “user” and “assistant” prefix can steer a base model toward the kind of continuation the user wants. Jeremy describes this as using familiar text patterns; it does not mean the model independently acquired a human identity.

A conversation template makes those roles explicit, but a base model can continue past a useful answer because it has not learned the desired stopping behavior. The interface feeds the growing sequence back into the model one token at a time.

Post-training starts with examples of the desired user-and-assistant exchanges. Training on responses and a stop token makes the model more likely to answer in the expected format and end its turn, turning a general text completer into an instruction-following assistant.

Examples of completed exchanges teach the assistant to generate an answer after a user message. A special end marker supplies a signal for where the response should stop.

A tokenizer can show the tokens and role markers sent to the model behind an ordinary chat interface. Matthew notes that people can experiment with open-weight models and a free Google Colab tier; the demonstration uses a T4 GPU.

Following instructions does not ensure correct reasoning. Jeremy shows an arithmetic problem that an assistant answers confidently but incorrectly. Prefilling or asking for a step-by-step solution can sometimes improve a result, though it does not guarantee correctness.

A request to think step by step can encourage the model to produce intermediate reasoning before an answer. Matthew recalls using such prompts during early model reviews, and Jeremy distinguishes publicly known techniques from what laboratories may have known privately.

The speakers distinguish a prompt that elicits a step-by-step response from training a model to improve its own attempts. Jeremy says public text often contains final answers rather than the process that produced them, making reasoning traces a separate training challenge.

Finished articles and programs generally show the result of people’s work, not every intermediate step. Jeremy argues that this makes it harder to learn problem-solving processes merely by imitating ordinary public text.

Jeremy describes reinforcement learning as training from trial and error, and points to DeepSeek R1 as a public example of such methods. He introduces group relative policy optimization as one approach to comparing multiple model responses to a problem.

For an illustrative math problem, Jeremy suggests asking for many independent attempts and grading the results. The training signal can then change the probabilities of future responses.

A simpler approach would generate many attempts, keep correct ones and train on them. Jeremy calls this rejection sampling. It can work, but he says it does not directly use negative examples and may reinforce incidental tokens in successful responses.

In the group approach, multiple answers are scored and compared with the group average. The relative score gives a signal for making the tokens in a better-than-average answer more likely and those in a worse answer less likely. Matthew clarifies that this is a post-training process.

A good relative score increases the likelihood of the corresponding generated response; a poor relative score decreases it. Jeremy notes that every token in an attempt receives a shared signal in the simplified explanation.

The model generates the examples it is trained on, an arrangement Jeremy calls on-policy training. He argues that using behavior the model can already produce can make adjustments less disruptive than imitating another system, while acknowledging that other capabilities can still degrade.

Jeremy contrasts learning from one’s own generated attempts with copying another source’s solutions. His claim about reduced interference is qualified: reinforcement learning can still degrade other capabilities.

For tasks with a reliable way to check outcomes, a model can repeatedly try, receive a score and improve. The quality of the verifier matters: the procedure needs a way to distinguish successful from unsuccessful attempts.

Matthew asks why reinforcement learning is not used from the beginning. Jeremy says researchers are exploring ways to merge stages. In his explanation, pretraining on broad outside data and later training an assistant serve different purposes.

After an assistant has been trained to follow a particular format, further training on an outside corpus can change its behavior and style. Jeremy offers code data as an example and says researchers are experimenting with combining reinforcement learning with other stages.

An assistant trained later on a corpus of code may start to reproduce that corpus’s style instead of maintaining its prior conversational behavior. Jeremy presents this as one challenge in mixing phases, not proof that mixed approaches are impossible.

Finally, Jeremy asks whether systems trained on existing material can produce new knowledge. He thinks reasoning over longer chains may let models build new abstractions and discover results not explicitly present in the data, while saying that time will test that prediction.

Jeremy compares reasoning with gradually building abstractions that make a previously unlikely conclusion reachable. He offers this as an intuition for why next-token systems could eventually arrive at new results.

Matthew closes by encouraging interested viewers to learn the fundamentals and experiment. He says a research career requires substantial work but is not limited to people with a PhD.