Lance Martin: Audit Prompts, Build Better Evals and Optimize Agent Cost

Anthropic’s Lance Martin argues that application teams often operate inside the available cost-quality frontier because of small configuration errors. Fixing those errors can improve quality and reduce cost at the same time.

Prompts accumulate debt like code. Martin recommends reviewing instructions written for older models, including blanket requests to think step by step, repeatedly verify work or be maximally thorough.

His claim is model- and task-specific: newer frontier models may spend unnecessary tokens on such rituals. Contradictory rules can also confuse a stronger instruction-following model. Remove an instruction only after measuring the effect on the actual task.

Martin demonstrates prompt-audit skills for Claude Code configuration and API applications. They look through project instructions, tools and prompts for patterns that may no longer help.

On a synthetic evaluation of roughly 50 customer-support tickets, his example compares a legacy prompt, a model upgrade and an audited prompt. The audit improves the reported score and lowers cost for that benchmark; it is not a guarantee for every workload.

Next comes prompt caching. Agent conversations repeat much of their context over many turns, so reading cached tokens can reduce expense and latency. Martin notes that the exact pricing depends on the model and should be checked.

Caching requires a matching prefix. A change early in the request can invalidate the reusable prefix, so put stable system instructions and tools before project, session and per-turn material where the API permits.

He recommends monitoring cache-hit rates and using diagnostics to identify prompts that can be reorganized. In internal benchmark harnesses, his team reports cost reductions of up to roughly 72% with no meaningful measured performance change after correcting cache use.

For optimization, Martin says a useful evaluation must be climbable: stronger models and more effort should generally help, the best model should have room to improve, and repeated runs should have low variance.

Ambiguous or impossible cases can create illusory headroom. A team may spend tokens optimizing toward a score that no reasonable solution can achieve, so it should inspect the data and grading criteria before running a search.

He describes a Claude Code skill that helps construct an evaluation from traces or other data. The user still decides what matters, inspects examples and checks whether the grader reflects the desired behavior.

Once the evaluation is sound, an agent can iteratively change a prompt or skill and measure the result. Martin calls this hill climbing, but warns that improvements can reflect overfitting to the evaluation rather than production performance.

For example, an optimizer may add an OCR tool because one benchmark case needs it even though real users do not. Distinctive wording in benchmark examples can similarly encourage prompt changes that do not generalize.

His approach separates training and test examples. The optimizer examines failures in the training split, while scores on the held-out split give an early warning if training improves and test performance stays flat.

Martin describes improving a Claude API documentation skill that initially scored 66% on his evaluation. He let the system inspect failures and revise the skill against the documentation; the result is an example of a narrow, measurable surface.

The same process can search for lower cost while keeping performance at least at baseline. It can try combinations of model, effort and prompt, rejecting configurations that degrade the evaluation.

In a customer-support example, Martin reports a roughly 90% cost reduction with higher performance after changing model and effort and refining the prompt. This is a reported result for that particular synthetic benchmark.

His rule of thumb is to test a stronger model at lower effort: it can use fewer total tokens than a weaker model working at high effort. The best choice remains empirical and depends on the task and pricing.

Long-running agents require a different harness from short chats. Martin focuses on recoverability and security when an agent may run for hours or days.

He describes separating the agent harness, sandbox and append-only session log. If a sandbox or harness fails, the orchestration layer can recreate it and recover the session from independently stored state.

For secrets, Martin says managed agents use a vault so the model can refer to credentials through tools without seeing their contents directly. He presents this as one control for long autonomous runs.

His recap connects the pieces: audit prompts, use the cache, design good evaluations, guard against overfitting and choose a recoverable, secure harness for long-running work.

Asked how much evaluation design can be automated, Martin says Claude can scan traces for candidate failures. He wants the workflow to show raw examples to a human before turning them into evaluation cases.

Human feedback can become an evaluation signal for hill climbing, provided it distinguishes better from worse outcomes. Martin emphasizes a collaboration in which people set goals and inspect the data while the model assists with repetitive work.

Prompts and skills are convenient optimization surfaces because they are easy to edit and directly influence behavior. Modifying a whole harness can work too, but larger code changes are harder to interpret.

That convenience comes with a warning: prompts can overfit easily. Even a train-test split is not a substitute for representative examples and careful checks against real production behavior.

For a repeated workflow, Martin suggests asking the model to turn the procedure into a reusable skill. For optimization, begin with a simple baseline skill and improve it against a defined evaluation.

Open-ended tasks are harder to grade than tests with deterministic answers. Martin recommends beginning with a rubric a person can understand and, where possible, using clear binary checks instead of vague numerical ratings.

He suggests checking a rubric across multiple models and repeated runs. Pairwise comparisons can also help: deciding which of two artifacts is better may be more reliable than assigning one an arbitrary score.

An evaluation distribution should reflect cases that matter in production. Sampling only failures of one model can bias the benchmark toward that model’s particular weaknesses, which another model may solve for incidental reasons.

Martin clarifies that multiple rating categories are possible when their meanings are well specified and graders agree. Binary grading is a simplifying heuristic, not a requirement for every evaluation.

Whether a personal automation needs an evaluation depends on the stakes. A workflow producing important external deliverables or handling sensitive work warrants more rigorous checks than a low-impact convenience tool.

Hill climbing is not guaranteed to converge. If it stalls, Martin says the optimizer can cluster remaining failures and look for ambiguous instructions or faulty graders, rather than spending indefinitely on impossible cases.

He says the hill-climbing skill is open source and might be adapted to other frameworks, while noting he has not personally checked all integration details. The underlying loop is general: propose a change, measure it and retain useful changes.

On effort settings, Martin recommends trying a capable model at low effort first and increasing it when measurements justify the extra work. A small model at high effort can cost more in tokens without producing the best result.

An optimization run also needs a budget. A poorly specified evaluation can consume many tokens pursuing ambiguous cases, so spending limits and evaluation quality work together.

Humans determine the goal and decide what matters; Claude can then inspect traces and flag likely failures. People review those candidates before promoting them into an evaluation set.

Martin says evaluation ownership follows product ownership rather than belonging exclusively to one specialist team. Tools can reduce setup work and make it easier for people in different roles to participate.

For generative interfaces, he has seen pairwise judges compare variants across many trials and aggregate results. Similar model-based grading can be used for retrieval and writing when rubrics are carefully designed.

He connects the method to autoresearch: both use a measurable objective, run experiments and adapt based on results. The specific surface and objective differ, so a workflow may need to be adapted.

Multiple objectives such as cost, quality and speed can be considered together, but each added objective makes the search harder. Martin suggests that changing one dimension at a time may sometimes help.

Asked how far back prompt-audit advice applies, he says the boundary between older and newer models is imprecise. The skill includes some model-specific guidance, and teams should test changes when migrating.

He explains effort as guidance about how hard the model should think, not a simple hard cap. Difficult tasks can make high-effort settings much more expensive, so teams should compare outcomes and token use.

The method resembles training in its use of objectives and train-test splits, but Martin distinguishes it as a runtime change to prompts or skills after deployment.

Evaluation awareness is a broader concern, but in this setting he sees overfitting as the more immediate practical problem. A model can improve a benchmark number by exploiting peculiarities that do not carry into production.

Small evaluation sets can be noisy, unrepresentative and easy to overfit. Martin does not give a universal minimum size; he recommends considering the signal and task when choosing the data.

For very long tasks, each optimization round takes longer and failures become more costly. Reliable, recoverable infrastructure becomes necessary when a search runs for many hours or days.

Agent evaluations may need multiple turns and can grade the entire trajectory, the final outcome or both. Traces are useful inputs for building such cases, but the right grading design depends on the task.

For context reliability, Martin returns to keeping the session log independent of the harness. That allows an agent to retrieve older material after compaction rather than losing information that might later matter.

Open-ended and nondeterministic tasks put more pressure on evaluation design. Clear objectives, understandable rubrics, agreement across graders and low variance give an optimizer a more useful signal.

As models improve, Martin expects a healthy evaluation to show corresponding progress; saturated or ambiguous benchmarks may need replacement. Designing challenging evaluations remains an ongoing task.

He describes a production feedback loop: monitor traces, select meaningful failures for an offline evaluation, optimize against them and keep solved cases as regression tests. The hard part is deciding which production outcomes truly matter.