When to Optimize an AI Agent’s Prompt or Its Weights

Michael Ryan compares prompt search, reinforcement learning and distillation, starting with the evaluation that makes any improvement measurable.

MRMichael RyanPonente · Stanford University
Canal
Stanford Online
Publicado
17 nov 2026
Vídeo
1:20:34
Lectura
11 min
Diapositivas
4
Original
inglés
Abrir en YouTube
Al desplazarte, el vídeo se reduce a la esquina inferior derecha y sigue reproduciéndose. Los subtítulos siguen tu idioma de lectura.
Idioma de lecturaDisponible en 2 idiomas
Este artículo aún no está en español, así que se muestra el original en inglés.
RESUMEN

En 30 segundos

Before changing a prompt or training a model, build an evaluation that captures the behavior your agent needs. Stanford’s Michael Ryan shows how prompt optimizers can keep successful examples and use failure feedback to revise instructions without changing model weights. Weight training instead can turn large numbers of trajectories into a smaller model, while reinforcement learning rewards the student’s own attempts. The choice depends on access, rollout cost and how often the task changes; the two approaches can also be combined.

  1. 1An evaluation defines what optimization can improveRyan says a system, task data and a metric are all needed. An email extractor’s tests can capture requirements such as date format that a short prompt omitted.
  2. 2Successful traces can become examples without changing weightsDSPy-style bootstrap search retains good outputs, tries combinations in context and selects based on evaluation.
  3. 3MIPROv2 searches instructions together with demonstrationsThe method uses program and training-data summaries to propose instructions, then uses search to compare candidate combinations.
  4. 4GEPA keeps prompts with complementary strengthsThe optimizer reflects on traces to mutate prompts and can merge useful components from different candidates.
  5. 5GRPO compares multiple attempts instead of training a value modelA group of solutions to the same task supplies a relative reward baseline, reducing some PPO machinery.
  6. 6On-policy distillation coaches the student on its own movesThe student generates a trajectory, while the teacher provides token-level probability information for the states the student actually reaches.
  7. 7The practical choice depends on cost, access and dataRyan recommends context optimization for API-only models or expensive rollouts, and weight optimization for a cheaper student or shorter context.

Conclusiones

  1. Start with representative data and a measurable evaluation; optimization follows whatever the metric rewards.
  2. Try context optimization when a strong model is available only through an API or the task changes frequently.
  3. Use weight training when a cheaper student, a shorter context or many training examples justify the additional work.
  4. In GEPA, retain prompts that solve different examples and use observed failures to propose revisions.
  5. On-policy distillation scores the student’s own trajectory with a teacher, rather than only copying teacher-generated examples.
TEXTO COMPLETO

Lee el texto editado por capítulos

Solo la transcripción
01

Why optimization starts with an evaluation

Examples and a measurable goal let a team adapt a general model to its actual task.

When building this lecture, I asked Codex to convert slides from an earlier talk into our course format. It reverse-engineered Keynote's file structure and wrote the conversion code, which impressed me.

Consider three prompts for solving math problems with PaLM 2-L. In the experiment, “take a deep breath and work on this problem step by step” did better than the alternatives. The point is not that this phrase is universally best: you would not reliably know the answer without testing the model and task.

Optimization lets us specify intent using data and an evaluation rather than encoding every requirement manually. We need three things: a system, examples or feedback, and a metric that tells us whether a change helped.

Three ingredients for optimization
DIAPOSITIVA 1 / 4Three ingredients for optimizationData, the system and an evaluation or metric are all needed to measure improvements.

Does stronger model capability make prompt tuning unnecessary? Small wording tricks may matter less, but context optimization still helps with the last mile: a general model must adapt to the precise job a product needs it to do.

An evaluation can express details missing from an instruction. An email-to-calendar extractor may need a parseable ISO date format and a policy for multiple events, even when the written prompt merely says to extract dates and times.

Optimization can target reliability, domain performance, cost or latency. It can also help a team adapt when the underlying model changes: retain the evaluation, then search for a configuration that works with the new model.

02

Searching for useful demonstrations

Few-shot methods test example selections and can bootstrap successful traces.

Few-shot prompting puts several examples alongside the task description, such as English phrases paired with French translations. The model learns from that context without changing its weights.

An early paper found that the order of demonstrations affected model performance. Its authors tried permutations and measured the distribution of predictions on a probing set, without using that set's ground-truth labels to choose an ordering.

Global entropy in that paper measures variation across predicted labels: a model that always returns the same class has low entropy. Local entropy concerns how evenly a model distributes probability across labels for a particular example. In the results, the global measure tended to select stronger orderings.

The main lesson is the search pattern: try different examples or arrangements and observe how they change performance. The paper found improvements over random orderings, though the precise best prompt depends on the model and task.

Labeled examples often give an answer without showing the reasoning. The STaR approach generates a rationale and answer, keeps successful examples, and can give a hint containing the right answer when trying to generate a rationale after an error.

DSPy uses a related bootstrap idea without requiring a weight update. Run training inputs, retain successful traces, place combinations in the prompt, and evaluate the resulting system. A random search can pick six examples from a pool, test them, then try another six.

03

Let models propose improved instructions

OPRO and MIPROv2 use past performance and task context to generate and search prompts.

Reflective optimization asks what went well or poorly in a run and uses that information to propose a revision. OPRO gives a model a history of candidate instructions and their scores, then asks it to suggest a new instruction to test.

The prompt behind the opening poll came from that OPRO setting. The researchers compared generated instructions using the same target model; in practical API evaluations, multiple trials help account for nondeterministic outputs.

Why might a model-written prompt work? One intuition for older models is that it may resemble their training distribution. Today I see a more important use: failure traces can reveal missing task requirements, such as the required date format.

MIPROv2, work I helped lead, combines three steps: bootstrap successful demonstrations, propose instructions informed by the task, and search combinations of instructions and examples. Rather than sampling every combination uniformly, it uses a Bayesian search.

An iris classifier as a language-model program
DIAPOSITIVA 2 / 4An iris classifier as a language-model programThe example uses flower measurements to illustrate bootstrapped demonstrations.

In the iris example, the program receives petal and sepal measurements and predicts a species. Run training instances, score the prediction, discard unsuccessful traces and retain correct ones as possible demonstrations.

To propose instructions, the optimizer summarizes the program, inspects examples from the training data and considers successful traces. It does not feed the held-out development examples into this instruction-generation step.

MIPROv2 also used prompting tips such as asking for a high-stakes scenario or a persona. Some tips were effective at the time, but those specific tricks have not necessarily aged as well as the broader approach.

The optimizer samples an instruction and a set of demonstrations, runs the program on a batch and updates its estimate of which choices are useful. Periodic full evaluations reduce the chance that a noisy batch determines the final pair.

A correct answer does not prove the accompanying reasoning is correct. We did not generally monitor reasoning traces in that work; an evaluation could include a judge for reasoning quality, but optimizing an imperfect metric can produce odd behavior or reward hacking.

MIPRO still returns a selected instruction-and-example combination rather than the whole distribution. The search can be viewed as a multi-armed bandit problem, though other ways of using the distribution might be interesting.

04

Reflect on failures and preserve complementary prompts

TextGrad and GEPA use textual feedback; GEPA maintains candidates that succeed on different cases.

TextGrad describes feedback flowing backward through a chain of language-model calls. A model reads what happened and writes textual suggestions for earlier stages. Despite the borrowing of deep-learning terms, this process does not compute an actual derivative through the text.

GEPA, Genetic-Pareto Prompt Evolution, keeps a pool of prompt candidates and evaluates them on individual tasks. A candidate that succeeds on a case others miss can remain valuable even if another prompt is stronger elsewhere.

The method filters toward a Pareto frontier: a prompt dominated by another that performs at least as well on the relevant cases need not be sampled. The pool retains complementary candidates so the search can build on different strengths.

One way to create a candidate is reflective mutation. Sample a prompt, run it on a batch, inspect the outputs and metric feedback, and ask a model to revise the prompt. This uses the content of a failure, not only its numerical score.

Another operation merges components from two candidate programs. In a multi-stage system one prompt might work well for extracting an email and another for selecting an emoji; a system-aware merge can combine the useful parts.

A new candidate first faces a batch evaluation. If it improves, it is evaluated more broadly and added to the pool; otherwise it is discarded. The process repeats within a fixed budget.

A student asks whether starting from one seed can trap the search in a local optimum. I suggest that seeding with several independently generated instructions, as in other methods, could be explored; I do not know whether that experiment was tried in this work.

The lecture's privacy example rewrites a sensitive user query with a local model, sends a sanitized request to a powerful external model, then combines the external response with the original query locally. The purpose is to obtain useful reasoning without sending the private details.

GEPA evolves the first and final prompts in that system. Candidate revisions add guidance on identifying sensitive information, structuring the transformed request and explaining the transformation; different branches improve different aspects of the task.

On the benchmark table shown in the lecture, GEPA and its system-aware merging variant generally perform strongly. The larger pattern is to combine useful demonstrations with reflection on real task behavior, while remembering that the evaluation defines what the search rewards.

Asked about population diversity, I note that the retained pool is constrained by the frontier over evaluation cases. There may be useful ways to alter the algorithm to preserve a wider range of candidates.

05

Train weights from teacher trajectories

Supervised fine-tuning can transfer agent behavior into a smaller model.

Weight optimization changes the model rather than only its context. Supervised fine-tuning trains on desired outputs by increasing the probability of the correct next tokens through a cross-entropy loss.

For an agent, a capable teacher can complete tasks in different environments. The student's training examples contain the task and the teacher's trajectory: commands, tool results, reasoning and final answer. Depending on the pipeline, trajectories may be verified or filtered.

One example asks an agent to find matching files while excluding a subdirectory and capture both standard output and error. The teacher writes a command, inspects the result and verifies what it captured; that sequence becomes a potential training example.

OpenThoughts-Agent combines tasks from multiple sources, chooses a teacher, generates trajectories and filters them. A teacher is not always best merely because it is the strongest available: a smaller student may learn more effectively from behavior closer to its own capabilities.

The paper's plots show performance improving as training data grows across the demonstrated tasks. Weight tuning is relatively data-hungry, but supervised training can make use of many examples when good data and a suitable teacher are available.

FireAct mixes several reasoning patterns, generates trajectories with a teacher, filters incorrect results and reformats successful ones for agent fine-tuning. In its reported comparison, a fine-tuned smaller model exceeded a prompted baseline on the task shown.

Students ask whether this is distillation. It is often called hard distillation: the student learns from the teacher's generated outputs. Other forms can use more information from the teacher's probability distribution.

06

Reward the student’s own attempts

Ryan compares PPO and GRPO as approaches to reinforcement learning for agent tasks.

Reinforcement learning instead samples the student's own attempts and rewards outcomes. For coding, tests can provide a verifiable signal; where direct verification is unavailable, a model judge may be used, but the training process still needs a reward.

In the file-search example, compare the output file with the expected result and reward a correct match. Vary tasks, environments and difficulty so the student practices its own actions under different conditions.

Proximal Policy Optimization, or PPO, uses a policy model being trained, a reward signal, a reference model and typically a value model. The reference helps discourage excessive drift; the value model estimates how useful positions in a trajectory are.

The PPO illustration combines the outcome reward with the difference from the reference and a per-token value estimate. An advantage estimate then helps adjust the policy toward actions judged better than alternatives at that point.

The cost is substantial: several models must be held in memory, the policy and value model are updated, and value calculations run across tokens. A reward can come from an environment rather than a separate model, but the remaining machinery is still expensive.

GRPO, introduced in DeepSeekMath, samples a group of answers for the same task and scores each. It removes the separate value model and estimates relative advantage from the distribution of rewards within that group.

GRPO compares rewards within a group
DIAPOSITIVA 3 / 4GRPO compares rewards within a groupThe group mean and standard deviation give a normalized advantage for each sampled output.

For example, if one code solution passes seven of ten tests and the group average passes four, that solution is above average. Subtract the group's mean reward and divide by its standard deviation to obtain a normalized advantage.

The GRPO objective uses a ratio between current and previous policy probabilities, weighted by that relative advantage. Clipping limits unstable updates, and a reference-model term can constrain how far the policy moves.

DeepSeek R1 illustrates a practical combination: generate reasoning behavior with reinforcement learning, collect examples for supervised training, then apply further training. The lecture compares GRPO and PPO on the paper's math tasks rather than treating one result as universal.

Supervised training can bootstrap a student's ability to start a task; reinforcement learning can then expose it to more varied environments. If reinforcement learning performs worse than an available supervised approach in a particular setting, there is no reason to use it just for its name.

07

A teacher can score the student’s own trajectory

On-policy distillation combines student-generated behavior with a denser teacher signal.

On-policy distillation offers another choice. Watching grandmasters play chess is not the same as playing your own game with a coach who responds to your moves; the latter feedback applies to situations you actually reach.

With ordinary supervised training, the teacher generates the trajectory. With reinforcement learning, the student acts and may receive only a final score. On-policy distillation lets the student act while a teacher scores its choices using token probabilities, producing a denser signal.

In the lecture's example, a student solves an arithmetic problem while ignoring that ice cubes in a hot pan would melt. A teacher's token-level probabilities can penalize the mistaken assumption as the student's own answer unfolds.

Ryan compares a Qwen example in which off-policy distillation, reinforcement learning and on-policy distillation achieved different results and costs. In the reported experiment the on-policy method reached the highest score with fewer GPU hours; that is a result for the setup shown.

08

Use an agent to run the search

Self-modifying harnesses and autoresearch turn code changes and experiments into a loop.

The Darwin Gödel Machine treats an agent's own harness as code it can modify. It keeps an archive of versions, selects earlier candidates for further changes, evaluates them and branches. Exploring older variants matters as well as improving the latest one.

In autoresearch, an agent works in a repository, proposes code changes, runs experiments and keeps or reverts them according to a metric. The evaluation is the control surface: if the metric is poor, automated iteration may optimize the wrong thing.

09

Choose context, weights, or both

The right choice depends on access, rollout cost, prompt length and available training data.

Prompt and weight optimization can work together. One study alternates prompt optimization, weight training and another prompt pass; the lecture also contrasts GEPA's sample efficiency with weight training's ability to benefit from larger datasets.

Weight optimization and dataset size
DIAPOSITIVA 4 / 4Weight optimization and dataset sizeThe shown OpenThoughts-Agent results compare performance across training-set sizes.

Choose context optimization when a strong model only offers API access, when the task changes frequently or when environment rollouts are expensive. Consider weight training when small models do not respond well to prompting, a teacher can produce a cheaper student, or a growing prompt becomes too long.

Más de este canal

6 diapositivas58:52EN
Stanford Online· 17 nov 2026Cynthia Bailey: Define What an AI Agent Can Do and When It Must StopCynthia Bailey · Stanford UniversityConnectors let models act through other apps; explicit permissions, human checks and stopping conditions determine how safely they do it.AI agentsMCPIngeniería de agentes+2
1:09:27EN
Stanford Online· 17 nov 2026Cynthia Bailey: Keep Testing AI After You ShipCynthia Bailey · Stanford UniversityOld test data can miss changed user behavior. Bailey shows how independent review, post-launch checks and better data expose AI risks.Agentes de programaciónAI SafetySeguridad y gobernanza+1
11 diapositivas1:18:55EN
Stanford Online· 16 nov 2026Michael Ryan: More AI agents can make a task harderMichael Ryan · Stanford UniversityParallel agents can explore more, but coordination costs and conflicting work can make a single agent more effective.Multi-agent systemsAgent orchestrationAgent memory+2
8 diapositivas1:18:54EN
Stanford Online· 15 nov 2026AI Agent Memory: What to Store, Retrieve and ForgetMichael Ryan · Stanford UniversityMichael Ryan compares agent memory architectures and shows why remembering more does not always improve decisions.Agent memoryAI agentsGeneración aumentada por recuperación+2
9 diapositivas1:16:51EN
Stanford Online· 14 nov 2026Michael Ryan: Choosing Agent Patterns by Control Flow and ContextMichael Ryan · Stanford UniversityStanford’s CS329Z explains workflow patterns, reasoning-action loops, and ways to keep an agent’s context focused.Ingeniería de agentesAI agentsFlujos de desarrollo y pruebas+1
4 diapositivas1:17:14EN
Stanford Online· 13 nov 2026DSPy, LangGraph and the engineering around an AI agentMichael Ryan · Stanford UniversityMichael Ryan compares agent frameworks through a calendar demo, then shows why evaluation, recovery and tracing matter.Agent orchestrationEvaluacionesIngeniería de agentes+2