Why reading a skill file cannot reveal everything an agent will do

SUMMARY

In 30 seconds

A skill can retrieve instructions or execute code outside its entry file, so static inspection alone can miss behavior. Zenity researcher Roey Ben Chaim describes running skills in tailored sandboxes and recording their side effects. His team reports that about 100 of thousands of exercised skills were removed after review, and recommends inspecting dependencies, restricting access and monitoring execution.

1The same behavior can be legitimate or suspiciousAn official diagnostic skill accessed configuration and metadata that an information stealer might also seek.
2The entry file may not contain the full payloadDependencies and remote instructions can change what actually runs.
3Agent traces alone miss some side effectsThe team also captures process, filesystem and network activity.
4The team reports roughly 100 removals after testing thousandsThe figure is from its own investigation and should not be treated as an ecosystem-wide rate.

Takeaways

Inspect what a skill loads as well as its entry file.
Observe process, file and network effects in an isolated test environment.
Interpret suspicious-looking actions in the context of the intended task.
Limit a skill’s access and treat it like a software dependency.

TopicsAI security Agent engineering Security & safety Coding agents

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

Why static skill inspection has limits

Skills may load code and instructions from elsewhere. Legitimate diagnostics can also resemble malicious behavior.

An agent skill can include instructions, resources and executable code written by someone else. Roey Ben Chaim of Zenity describes testing thousands of skills in an isolated environment to observe what they actually do.

An official Azure diagnostic skill read cloud and Kubernetes configuration and contacted identity and metadata services. Those actions can resemble credential theft, illustrating why behavior has to be interpreted in the context of the task.

Reading only the entry-point skill file can miss instructions or code retrieved from elsewhere, including installation dependencies. Ben Chaim also describes an example where misleading comments fooled an LLM-based analyzer.

Exercise skills in a tailored sandbox

The team builds environments and task scenarios, then observes agent and operating-system side effects.

The team built a sandbox that runs an agent with a skill and observes the trace. A first problem was missing context: a skill might need a particular CLI, project or service. A world builder prepares an environment suited to the skill.

SLIDE 1 / 2A tailored environment for skill testingThe diagram adds an environment builder before executing the skill and assessing its trace.

Different skills need different tasks to exercise their behavior. An operator selects scenarios to explore relevant execution paths. The researchers say they also encountered refusals and used an adversarial operator in their controlled security research.

Agent messages alone miss some side effects. The team therefore captures process creation, file activity and network traffic, including activity spawned by package-installation scripts.

What the investigation found

Ben Chaim describes a proof of concept, reported removals and a credential-harvesting campaign.

A proof-of-concept skill installed another skill that retrieved instructions remotely. Ben Chaim says observing the whole chain exposed behavior that a scan of the initial file could miss.

SLIDE 2 / 2The reported skills.sh investigationThe slide summarizes the number of skills exercised and removals after review.

He reports that the team exercised thousands of skills and about 100 were removed after review for malicious or dangerous behavior. This is the team’s reported finding, not a measured prevalence for every skill ecosystem.

He describes a lookalike package connected to a nonofficial registry and an information-stealing payload. Other observed patterns included persistence instructions, silent replacement of another skill, remote prompts and hooks that trigger repeatedly.

The team also reports a campaign that imitated an agent-orchestration project and fetched a credential-harvesting payload. Ben Chaim says the researchers reported it to platform providers, who removed the identified material.

Treat skills as dependencies

Inspect loaded content, limit access and monitor execution.

His conclusion is to treat skills as dependencies: inspect what they load, restrict access and observe execution. The wider supply chain also includes any external content that enters an agent’s context, from websites to package data.