30초 요약
A skill can retrieve instructions or execute code outside its entry file, so static inspection alone can miss behavior. Zenity researcher Roey Ben Chaim describes running skills in tailored sandboxes and recording their side effects. His team reports that about 100 of thousands of exercised skills were removed after review, and recommends inspecting dependencies, restricting access and monitoring execution.
- 1The same behavior can be legitimate or suspiciousAn official diagnostic skill accessed configuration and metadata that an information stealer might also seek.
- 2The entry file may not contain the full payloadDependencies and remote instructions can change what actually runs.
- 3Agent traces alone miss some side effectsThe team also captures process, filesystem and network activity.
- 4The team reports roughly 100 removals after testing thousandsThe figure is from its own investigation and should not be treated as an ecosystem-wide rate.
핵심 정리
- Inspect what a skill loads as well as its entry file.
- Observe process, file and network effects in an isolated test environment.
- Interpret suspicious-looking actions in the context of the intended task.
- Limit a skill’s access and treat it like a software dependency.
챕터별로 다듬은 전문 읽기
Why static skill inspection has limits
Skills may load code and instructions from elsewhere. Legitimate diagnostics can also resemble malicious behavior.
An agent skill can include instructions, resources and executable code written by someone else. Roey Ben Chaim of Zenity describes testing thousands of skills in an isolated environment to observe what they actually do.
An official Azure diagnostic skill read cloud and Kubernetes configuration and contacted identity and metadata services. Those actions can resemble credential theft, illustrating why behavior has to be interpreted in the context of the task.
Reading only the entry-point skill file can miss instructions or code retrieved from elsewhere, including installation dependencies. Ben Chaim also describes an example where misleading comments fooled an LLM-based analyzer.
Exercise skills in a tailored sandbox
The team builds environments and task scenarios, then observes agent and operating-system side effects.
The team built a sandbox that runs an agent with a skill and observes the trace. A first problem was missing context: a skill might need a particular CLI, project or service. A world builder prepares an environment suited to the skill.
Different skills need different tasks to exercise their behavior. An operator selects scenarios to explore relevant execution paths. The researchers say they also encountered refusals and used an adversarial operator in their controlled security research.
Agent messages alone miss some side effects. The team therefore captures process creation, file activity and network traffic, including activity spawned by package-installation scripts.
What the investigation found
Ben Chaim describes a proof of concept, reported removals and a credential-harvesting campaign.
A proof-of-concept skill installed another skill that retrieved instructions remotely. Ben Chaim says observing the whole chain exposed behavior that a scan of the initial file could miss.
He reports that the team exercised thousands of skills and about 100 were removed after review for malicious or dangerous behavior. This is the team’s reported finding, not a measured prevalence for every skill ecosystem.
He describes a lookalike package connected to a nonofficial registry and an information-stealing payload. Other observed patterns included persistence instructions, silent replacement of another skill, remote prompts and hooks that trigger repeatedly.
The team also reports a campaign that imitated an agent-orchestration project and fetched a credential-harvesting payload. Ben Chaim says the researchers reported it to platform providers, who removed the identified material.
Treat skills as dependencies
Inspect loaded content, limit access and monitor execution.
His conclusion is to treat skills as dependencies: inspect what they load, restrict access and observe execution. The wider supply chain also includes any external content that enters an agent’s context, from websites to package data.
같은 채널의 다른 글
38:50EN
AI Engineer· 2026년 11월 10일LinkedIn’s self-improving agent first had to fix its judgesZiwei Guo, Saloni Gandhi · LinkedInA prompt-optimization loop improved a career agent, but unreliable evaluators required human calibration first.
12:48ENAI Engineer· 2026년 11월 10일Brian Douglas: Turn agent corrections into reusable review knowledgeBrian Douglas · Paper ComputeThe Paper Compute developer captures expert pushback in coding sessions and tests ways to reuse it during review.
12:20ENAI Engineer· 2026년 11월 10일Ayush Bhardwaj: Agent workarounds are feedback for your CLIAyush Bhardwaj · Allos AIAn audit of coding sessions shows where output, errors and stale instructions force agents to improvise.
12:57ENAI Engineer· 2026년 11월 10일Harvey’s agents scaled until their platform and merge queue brokeJoey Wang · HarveyAfter an alert spawned 900 agents and one engineer launched 950 tasks, Harvey added deduplication and fair scheduling.
23:00ENAI Engineer· 2026년 11월 10일A passing PR check needs evidence tied to the submitted codeStephen Chin, Cassandra Chin · Neo4jStephen and Cassandra Chin show how Tracetoon connects agent traces, code changes and checks before writing a reviewable pull request.
22:49ENAI Engineer· 2026년 11월 10일When many coding agents share one codebase, delivery must handle concurrencyPatrick Debois · TesslPatrick Debois explains how shared planning, merge strategies and risk-based review can help teams coordinate agents.