How language-model design shapes context engineering for agents

Stanford instructor Michael Ryan introduces the second CS329Z lecture. He will explain enough about language-model design to help students build agents on top of those models.

He divides the stack into model engineering, adaptation and applications. This course focuses on applications, while drawing on the lower layers when they affect design choices.

The goal is to connect choices in training and inference to practical limitations, such as long-context behavior and model selection.

A language model assigns probabilities to token sequences. Ryan explains the probability of a sequence as the product of each next-token probability conditioned on preceding tokens.

In practice, those probabilities are often handled as sums of log probabilities for numerical stability. The model can also be sampled to generate a continuation.

Ryan contrasts discriminative models, which distinguish classes, with generative models that model a distribution over possible sequences. He notes that the terminology can have fuzzy boundaries.

To generate text, the model assigns probabilities across its vocabulary and selects a next token. The new token becomes part of the context for the next step.

Greedy decoding always chooses the most likely token. Top-k restricts selection to a fixed number of candidates; top-p uses the smallest set reaching a chosen cumulative probability.

Temperature changes the shape of the sampling distribution: lower values generally concentrate probability on the most likely choices, while higher values spread it out.

Beam search explores several candidate continuations, at an additional time and memory cost. Ryan says the right decoding strategy depends on the task and budget.

In questions, he explains that tokens are chunks from a fixed vocabulary and that temperature can also affect generated reasoning.

Ryan briefly places transformers in the history of language models, then focuses on the decoder-only architecture common in modern systems.

Tokens are mapped to vectors, and positional information is added so the model can represent order. These representations then enter attention layers.

The attention layer projects representations into queries, keys and values. Query-key scores determine how information from preceding positions is combined.

A causal mask prevents a position from using future tokens during training, matching the information available when text is generated one token at a time.

The resulting attention output is projected and combined with a residual connection. Ryan answers questions about the mask and the role of position information.

Multi-head attention performs several attention operations with different learned representations. A transformer block combines attention, a feed-forward component and residual connections.

Standard attention allows substantial parallelism during training, but its attention computation grows quadratically with sequence length. That makes long contexts expensive.

Ryan outlines linear attention as a different way to maintain a condensed state. Its memory footprint can remain fixed with respect to sequence length in the illustrated formulation.

The tradeoff is that details of earlier text must be represented in that compressed state. He shows long-context retrieval results where the illustrated linear models degrade more than a full-attention baseline.

Hybrid architectures mix attention types, using full or local attention in some layers to improve retrieval while retaining efficiency in others.

For agent builders, model architecture matters because long histories and inference efficiency can influence which model and harness work well for a task.

Training requires a loss function. Ryan uses next-token cross-entropy to show how a model is penalized for assigning low probability to the observed next token.

Pretraining data can begin as web pages. Pipelines extract text, filter language and quality, remove repetition and duplicates, then mix the result with other sources.

He shows an example filtering pipeline that retains a small fraction of the original material. Even after filtering, training examples cover a wide variety of topics.

Teams adjust data mixtures for their objectives. Ryan describes a mid-training or cooldown phase that can place more weight on domains relevant to later use.

Supervised instruction tuning uses examples of prompts and desired answers to make a model behave more like an assistant.

Human preference training can compare responses and train a reward model, but preferences vary and annotations can be inconsistent. Ryan also raises concerns about incentives and the conditions of data workers.

Reinforcement learning with verifiable rewards uses tasks where an answer can be checked, such as code tests or a math answer, rather than relying only on a preference model.

For coding agents, he describes creating environments from repositories, introducing or reversing bugs, and rewarding solutions that pass tests.

In discussion, Ryan notes that long-context extension and realistic agent environments can be parts of later training. He also discusses the promise and limits of synthetic data.

Inference has two broad phases: prefill processes the prompt, and decode generates the response. The phases can be limited by different hardware resources.

Ryan contrasts compute throughput with memory bandwidth. Prefill processes many prompt tokens together, while token-by-token decoding can be limited by moving data.

The key-value cache retains attention information needed for later tokens. It avoids recomputing earlier work, but for full attention its memory grows with the sequence.

Time to first token is a useful prefill metric; time per output token describes decoding. Optimizing one phase does not automatically optimize the other.

Speculative decoding uses a small draft model to propose several tokens, then checks them with the larger model. Accepted proposals can reduce the number of serial large-model steps.

Inference-time scaling spends more computation on a task. Chain-of-thought prompting and reasoning models can generate intermediate work before an answer, at additional cost.

Ryan describes reasoning-effort settings and training approaches that trade correctness against a token budget. The user can choose how much effort to spend on a task.

Other strategies sample several answers and use a verifier or majority agreement. The usefulness of those approaches depends on the task and how answers are checked.

Agents often need structured input and output so tools and model calls share a predictable contract. Constrained decoding can restrict output to a specified structure such as JSON.

Ryan gives a DSPy example that declares inputs and outputs and extracts fields such as a name, email and intent from a message.

He also describes using a model as a classifier or action scorer when the system needs to choose among a limited set of outputs, including tool calls.

Context engineering concerns the whole set of information supplied to an agent: instructions, conversation history, memories, tool results and files.

Putting everything into context can be costly and can reduce performance on long-context tasks. Ryan shows a retrieval-and-counting benchmark where accuracy falls as documents accumulate.

He recommends avoiding unnecessary tools and context, retrieving relevant material when needed, and testing context design against the actual task.

Appending new information over successive turns can preserve more of the existing key-value cache. Inserting material at the beginning can invalidate cached work.

Ryan advises evaluating choices such as formatting and tool selection empirically, rather than assuming one context layout is always best.

Compaction summarizes or removes material once context grows too long. He recounts a case where an important confirmation instruction was lost during compaction, leading to unwanted deletion.

Recursive language models offer another approach: keep a long prompt in an environment, let a model inspect parts of it, and call submodels on selected pieces.

Ryan concludes that understanding model architecture, training and inference can inform context design, even when an application mostly uses existing models.