Test what a skill teaches before cutting its instructions

Nikolay Petrov describes skills as reusable packages of instructions that turn repeated prompts into portable agent capabilities. Some provide new knowledge; others primarily steer a repeatable process. Both can be useful, but this talk focuses on skills intended to teach a model something it did not already know.

A basic evaluation compares the same defined tasks with and without the skill loaded. Petrov recommends tasks that can be checked as deterministically as practical. His illustration has the model solve three of five tasks without a skill and five of five with it; the comparison can also include token use and time.

Even when a skill improves end-to-end evaluations, its contents may need revisiting as models evolve or token costs matter. Trying every combination of lines is expensive, while deleting text based on intuition can remove a detail the model actually needed.

Petrov proposes splitting a skill into small, independently checkable items, drawing on the idea of atomic facts from FActScore. Each item gets a question and an expected answer. An item can describe a general proposition or a more specific behavioral contract.

A skill might yield 25 or 30 such items. For each question, the team runs two arms—one with the skill and one without—and uses a model judge or panel of judges to assess the answers. Petrov recommends a judge from a different model family where practical and questions that do not reveal their own answers.

The test can run without tools to probe what the model already knows, or with tools such as web access to better reflect a practical environment. These are different measurements: external information can change what the model can answer without the skill.

The paired outcomes form a four-way map. If the model answers an item correctly both with and without the skill, it already has enough knowledge to answer that question in the tested setting. If it succeeds only without the skill, adding the skill appears to hurt that item.

If the model fails without the skill but succeeds with it, the item represents the desired gain in that test. If it fails in both conditions, the subject may still matter, but the skill as written did not make the tested model answer successfully.

Petrov says performance-degrading items were rare in the popular skills his group examined: he points to one case among nearly 400 items in the Anthropic skills shown. He reports seeing more such cases in some smaller, less popular skills; the finding is specific to the tested skills, models and questions.

The outcomes can be visualized against the skill text to show what is helpful, already known, harmful or still missing. Petrov cautions that the unit is an item, not necessarily a line: a single line or sentence can contain items with different outcomes.

For the reported experiment, Petrov says testing 29 skills cost about $14 in Sonnet and Haiku calls, with Sonnet used as judge. The cost is a figure for that run, not a universal price for evaluating every skill.

The evaluation can also change how strictly it scores an answer. A lenient check asks whether the general idea is present; a strict check asks for specific keys, parameter values or exact steps. Knowing a concept is not the same as knowing the particulars an agent must use.

In one comparison Petrov reports that Opus answered about 70% of skill items under a general understanding criterion but 22% under a specific-detail criterion. He interprets the difference as a reason to preserve exact data and operational details even when a frontier model knows the broad concept.

Across the models and skill collections shown, smaller models generally knew less of a skill than larger ones, but results also varied significantly by skill. Some skills are meant to steer behavior rather than supply novel facts, so a low novelty measure is not automatically a defect.

Novelty is an initial direction for optimization, not a substitute for task evaluations. Petrov suggests removing some information that appears already known, then rerunning the end-to-end evaluations. If task results hold, the shorter skill may save tokens and context; if not, the supposedly redundant material may still be useful.

The experiments used Documate, which Petrov describes as a framework for comparing agent context such as skills, AGENTS.md and MCP tools. He says it also supports evaluations of context impact, routing, descriptions and names, with the goal of making skill changes based on measurements rather than guesswork.