Most people who install a third-party skill pack into their agent press Enter on the strength of a README and a star count. How dangerous that actually is has been a matter of intuition, not measurement.
Not anymore. Tencent Zhuque Lab published an authorized injection assessment of DeepSeek Harness inside the Tencent/AI-Infra-Guard repo — 14,560 real agent runs. When content enters the agent through load_skill, indirect prompt injection fully succeeds 15.2% of the time. The same attacks delivered through read_document succeed 3.9% of the time. Installing a skill is 3.9x riskier than reading a document.
That number comes from A.I.G's own Research/ directory, not its marketing page. Which is exactly why this project deserves a source read rather than a README skim.
Five blades in one repo
A.I.G (AI-Infra-Guard) is the open-source AI red teaming platform from Tencent Zhuque Lab, a security research unit founded in 2019 inside Tencent's Security Platform Department. As of 2026-08-23: 5,477 stars, 518 forks, 26 open issues, latest release v4.5.2 shipped 2026-08-17, Apache-2.0. The main branch holds 6,397 files, split as Python 3.51 MB / TypeScript 1.13 MB / Go 777 KB.
That language ratio is the architecture: Go orchestrates, Python scans, Vue/TS renders. aig-server (web UI on :8088) dispatches tasks over WebSocket to aig-agent executors, with round-robin load balancing added in v4.1.14.
The line that actually matters is at the bottom of that diagram: A.I.G's rules come in two forms — YAML, and Markdown prompts. The first is reproducible, diffable, and gated by cmd/yamlcheck in CI. The second is executed by an LLM, and the same input does not guarantee the same verdict. Only one of the five scan surfaces belongs to the first category. That boundary determines which results you can put in a compliance report and which are only leads.
Layer one: the only part that needs no LLM
AI Infra Scan is the most conventional and most solid layer. Fingerprint the target to learn what component and version it runs, then match that version against a vulnerability database.
Fingerprints are 146 YAML files with familiar nuclei-flavored syntax:
info:
name: ollama
author: 腾讯朱雀实验室
severity: info
http:
- method: GET
path: '/'
matchers:
- body="Ollama is running"
version:
- method: GET
path: '/api/version'
extractor:
part: body
group: 1
regex: '{"version":"(\d+\.\d+\.?\d+?)"}'
Vulnerability rules are simpler still — one CVE per file, the core being a single version-range expression:
info:
name: ollama
cve: CVE-2024-37032
severity: MEDIUM
rule: version < "0.1.34"
I counted data/vuln/: 2,014 YAML files across 116 directories (data/vuln_en/ is an equal-sized English mirror, so the README's "2000+ CVE rules" holds — just don't add the two together). The README quotes "130 components, 1,888 rules" as of v4.5.0; its component count is broader than my per-directory tally, the gap being components that have a fingerprint but no dedicated vuln directory (data/fingerprints/ holds 146). The distribution is worth a look:
| Component | Rules | Share |
|---|---|---|
| openclaw | 657 | 32.6% |
| praisonai | 112 | 5.6% |
| langflow | 111 | 5.5% |
| flowise | 97 | 4.8% |
| openwebui | 87 | 4.3% |
| mlflow | 79 | 3.9% |
| vllm | 72 | 3.6% |
| remaining 109 components | 799 | 39.7% |
A third of the vulnerability corpus is bet on one ecosystem. That isn't a flaw, it's a choice — it reflects where Zhuque actually invests (one v4.1.15 changelog line reads "openclaw vuln count 628 -> 655"). But if OpenClaw isn't in your stack, compute A.I.G's real coverage from the other 1,357 rules.
One more detail: both openwebui (87 rules) and open-webui (44 rules) exist as separate directories. v4.5.0 just fixed a sibling of this bug ("Remove duplicate fingerprint open-webui.yaml", "Correct info.name to match fingerprint names (case-sensitive)"). At two thousand rules, naming consistency becomes an engineering problem in its own right.
Layer two: regex supplies evidence, the LLM delivers the verdict
skill-scan is the most interesting idea in the project. It has been split out as a standalone PyPI package:
pip install aig-skill-scan
export LLM_API_KEY="your-api-key"
aig-skill-scan --repo /path/to/skill -m deepseek-v4-flash -o result.sarif.json
The key point is that it is not a static scanner.
utils/pre_scan.py does contain 13 high-risk regexes — curl|sh pipe execution, the 169.254.169.254 cloud metadata endpoint, ~/.ssh credential paths, the ignore previous instructions injection family, base64.b64decode followed by exec, crontab persistence, executables pulled from pastebins. But their output is not a verdict. It's a hint block:
⚠️ The static pre-scan found the following patterns that warrant special attention; please focus the audit on whether these behaviors are necessary and what risks they pose…
That text is injected into an actual agent holding seven tools: ls, dir, grep, read_file, base64_decode, thinking, finish. Calls use an XML protocol, one tool per turn, with compact.md compressing context when it fills. The auditor decides which files to read, how many passes to take, and when to stop, then calls finish to emit a Markdown report tagged against SkillTrustBench's T01–T09 taxonomy and converted to SARIF 2.1.0 for GitHub Code Scanning.
The tradeoff is explicit: regex answers "where is suspicious," the model answers "is this a problem." Because a skill's attack surface isn't at the syntax layer at all — a skill can be a single SKILL.md with zero lines of code, and its install instructions are its behavior. prompt/agents/code_audit.md says so directly:
Install instructions, initialization commands, and prerequisite steps in SKILL.md are equivalent to code behavior and must be audited to the same standard as code.
Verdicts come in three grades: malicious (clear attack intent), suspicious (a real vulnerability without clear intent), normal. The interesting one is malicious trigger #9 — "three or more concurrent suspicious signals aggregate to malicious." That encodes "individually harmless, jointly fatal" as a rule, and it's a judgment no purely static tool can make.
The one design you only find by reading the source
utils/text_decoder.py contains _recover_utf16_mojibake, handling an attack I hadn't seen addressed in any comparable tool.
The attacker writes instructions into a file as UTF-16 bytes. A human sees garbage. The model sees garbage. A static scanner finds nothing — but swap the decode path at runtime and the instruction comes back to life. A.I.G's answer is to take the already-decoded string, encode('utf-16-le') it back, and re-decode as UTF-8. Two acceptance criteria: ASCII readability gain ≥ 0.25, or the presence of Cf/Co category characters (format controls / private use) in the original. On success it reports reversible_mojibake — and feeds the recovered text back through all 13 regexes, so a curl|sh hidden inside mojibake still gets caught.
pytests/test_charset_smuggling.py guards that path. This is the kind of code you only write after this trick has actually been used against you.
The other half of the v4.5.2 claim, which I could not find in the source
The README's v4.5.2 entry reads: "Skill-Scan: .pyc bytecode bypass detection + charset smuggling defense." The second half holds — see above. For the first half I pulled all three Python packages (skill-scan, mcp-scan, agent-scan) at main HEAD 4908db1 (2026-08-21, after the 08-17 release) and grepped -rniE '\.pyc|bytecode|marshal|dis\.dis|decompile'. Every hit:
skill-scan/skill_scan/utils/pre_scan.py:91: _SKIP_EXTS = {'.pyc', '.pyo', '.pyd', ...}
skill-scan/skill_scan/tools/dir/dir_actions.py:8: _IGNORED_EXTS = {'.pyc', '.pyo', '.pyd'}
skill-scan/skill_scan/agent/agent.py:43: _TREE_SKIP_EXTS = {".pyc", ".pyo", ".pyd"}
mcp-scan/mcp_scan/utils/pre_scan.py:111: _SKIP_EXTS = {'.pyc', '.pyo', '.pyd', ...}
agent-scan/agent_scan/utils/file_utils.py:38: '.wasm', '.pyc', '.pyo',
Five hits, five skip lists. Nothing reads, disassembles, or inspects bytecode anywhere. And the third hit is worse than the first two: _TREE_SKIP_EXTS applies to the project tree shown to the agent, so a .pyc isn't merely unanalyzed — it never appears in the file listing the auditor sees. The auditor doesn't know the file is there.
In fairness, this capability may live in the closed part of the stack (the platform backend, or the hosted service at matrix.tencent.com). But set against the charset-smuggling item in the same release note — code present, tests present, criteria present — the two claims sit an order of magnitude apart in verifiability. Whoever writes the changelog should reconcile with whoever writes the scanner.
And one thing that doesn't add up
prompt/system_prompt.md is a generic coding-agent system prompt, reused nearly paragraph for paragraph: "your name is <{name}>", "answer in no more than 4 lines", "do not add unnecessary preamble or postamble", "make the minimum changes necessary", "after you finish a file, stop — don't explain what you did."
Every one of those constraints is right for a coding agent. For a security auditor they collide head-on with the downstream code_audit.md, which demands "file path + line numbers, code snippets, technical analysis, impact assessment, remediation advice, attack path" for every confirmed finding. Upstream says be terse; downstream says write the report; an instruction override is what mediates. You won't see this in an F1 of 0.98, but it's the kind of design debt that quietly costs you points in the long tail.
Layer three: the detection rules are themselves Skills
agent-scan is the most self-referential part of the project. Its 14 detection rules are each a SKILL.md:
prompt/skills/
├── agentic-supply-chain-detection/SKILL.md
├── authorization-bypass-detection/SKILL.md
├── cascading-failure-detection/SKILL.md
├── data-leakage-detection/SKILL.md
├── direct-injection-detection/SKILL.md
├── file-path-traversal-detection/SKILL.md
├── hardcoded-secret-detection/SKILL.md
├── human-agent-trust-exploit-detection/SKILL.md
├── indirect-injection-detection/SKILL.md
├── inter-agent-comm-security-detection/SKILL.md
├── memory-poisoning-detection/SKILL.md
├── owasp-asi/SKILL.md
├── tool-abuse-detection/SKILL.md
└── unexpected-code-execution-detection/SKILL.md
Writing audit rules in the format being audited isn't a coincidence, it's two faces of one judgment: if the Skill format is expressive enough to carry an attack, it's expressive enough to carry a detection. It's the same phenomenon I traced in Google entering the Agent Skills race — once a format wins de facto standard status, offense and defense both move in.
Adding a detection capability therefore costs one Markdown file (v4.5.0 added 4 at once; v4.5.1 added 5 more). The price is equally plain: recall depends entirely on the model executing the rule, and no static check will tell you whether your new rule overlaps an existing one.
Layer four: the MCP red team, and the shot it doesn't take
mcp-scan shares the agent skeleton with skill-scan but carries two extra tools: execute (run commands) and mcp_tool (actually call the target MCP server's tools). That's the basis for dynamic verification.
Beneath it sits a redteam/ submodule with three collaborating LLM roles:
| Role | Responsibility |
|---|---|
| Attacker Agent | Generates the next attack message from goal + history, emitting thought / message / attack_technique / reflection |
| Target Runner | Interacts with the target |
| Evaluator Agent | Judges on_topic, scores 1–10, decides is_successful |
Two strategies: Crescendo (build trust → probe boundaries → escalate → strike, four staged phases) and TAP (Tree of Attacks with Pruning — branch multiple variants per round, filter by on_topic, then keep top-k by score). Six predefined targets aligned to OWASP Agentic Top 10: data_exfiltration, indirect_prompt_injection, ssrf_via_agent, rce_via_tool, privilege_escalation, tool_poisoning.
Looks complete. Then you open its own README:
Target Runner: currently in source-analysis mode — it reuses mcp-scan's code-reading capability to gather repository context, and the LLM simulates the MCP server's response to the attack. It does not actually start an MCP process.
The attacker is an LLM, the target is an LLM, the judge is an LLM. No real MCP process participates anywhere in the chain. What this red team line currently demonstrates is not "can this MCP server be compromised" but "does a model, having read this source, think it could be."
Credit where it's due: that sentence is in their own architecture docs, in bold. The classic failure mode for security tooling is dressing simulation up as measurement, and A.I.G chose honesty here. But anyone shipping reports off it must know the boundary — especially since mcp_tool exists on the main scan path and simply isn't wired into the red team orchestration, which makes it easy to assume both run the same way.
Layer five: the jailbreak suite is a vendored fork
AIG-PromptSecurity/ holds 505 files under a directory literally named deepteam. It's a vendored copy of confident-ai's open-source red teaming framework deepteam — a changelog line reads "Update DeepTeam repo URL to confident-ai/deepteam", and the subtree carries its own Apache-2.0 LICENSE.
On top of it they extended the attack operators. Currently 29 single-turn + 10 multi-turn:
- Single-turn:
code_chameleon,context_poisoning,deep_inception,encoding(with a1z26, affine and other variants),flip_attack,ica,jam,math_problem,multilingual,past_tense,prefill,promisqroute,stego,super_user,system_override, and more - Multi-turn:
actor_attack,bad_likert_judge,best_of_n,crescendo_jailbreaking,goat_jailbreaking,linear_jailbreaking,many_shot_jailbreaking,pair_jailbreaking,sequential_break,tree_jailbreaking
Sixteen vulnerability dimensions (bias, excessive_agency, pii_leakage, prompt_leakage, unauthorized_access, robustness, …), evaluated over AdvBench, CNSafe and SafeBench.
The value here is integration rather than originality, which is fine as long as attribution is clear — and it is.
The 14,560 runs are worth more than the tool
Research/ holds two original studies, and deepseek-harness-security-assessment is the most valuable thing I've read in this repo.
Design: 1,120 base cases (16 carrier channels × text/file delivery modes × 35 payload objectives), each paired with a naive baseline plus 12 attack methods, for 14,560 agent runs against the real DeepSeek Harness runtime (model deepseek-v4-flash), with controlled source tools and tracked simulated sinks — no mail is sent, no shell is executed, no funds move. Two evaluators: J_R, a deterministic rule-based judge checking taint delivery, sink calls, argument matches, canaries and refusals; and J_L, an offline semantic LLM judge over saved traces.
The headline numbers are unremarkable: J_R 5.6% full / 2.0% partial; J_L 5.3% full / 7.3% partial. The slices are where the information is.
Finding one: skills are the most dangerous input channel.
| Source tool | Runs | J_R full |
J_L full |
|---|---|---|---|
load_skill |
910 | 15.2% | 13.7% |
fetch_url |
2,730 | 8.6% | 6.8% |
read_channel |
910 | 5.4% | 5.4% |
read_email |
2,730 | 4.1% | 3.3% |
search_knowledge |
910 | 4.1% | 4.5% |
read_document |
6,370 | 3.9% | 4.4% |
Installing a skill beats reading a document by 3.9x, and clearly beats "fetch a URL off the internet" too. The reason isn't subtle: a skill is loaded as instruction, a document is read as data. The model's default trust level for the former is simply higher — the attacker doesn't need to break anything, only to get installed.
Finding two: change the delivery form, change the world.
The unicode_hidden channel (instructions hidden in invisible characters) scores 0.0% J_R full success in text mode and 25.5% in file mode — the highest cell in the table. Their README explains why this isn't "rename a text file with a different extension": file mode genuinely generates or modifies a carrier in its native format (edit metadata in a real PDF, edit cells in a real XLSX, generate an .ics calendar, construct real HTML) and then runs the format-specific parsing path to get the model-visible content. Representation-layer behavior is not something a plain-text approximation can measure.
This matches what I measured on prompt-fence bypasses: homoglyphs and full-width characters — the tricks that look like bypasses — went 0/12 across three models, and the only family that actually punched through was invisible characters. A.I.G's data supplies the half I hadn't run: that punching power is concentrated entirely in file-parsing paths and is roughly zero for plain-text delivery. That relocates the defense — sanitize after extraction, not only at the input box.
Finding three: fake_completion is the single best technique. Forging a "task already complete" state frame reaches 17.0% J_L full success in text mode against a naive baseline of 5.7%. Next come obfuscation (13.6%) and escape (9.3%). This class of attack doesn't argue the model into violating a rule; it convinces the model that the current turn's constraints are already over — mechanically the same family as the history-rewriting bug I traced in scanning directions for agent context injection: what gets poisoned isn't the content, it's the model's read on its own situation.
Worth noting that the target here, DeepSeek Harness, is one of the big-vendor open harnesses I took apart last week. The upside of open harnesses shows up concretely: a third-party lab could run 14,560 authorized tests against the real runtime and publish the results with the dataset attached. That cannot happen against a closed harness.
The second study, SkillJack (persistent skill backdoors in self-evolving agents), ships 65 disguised poisoned trajectories (20 data exfil + 15 privilege escalation + 15 unauthorized transfer + 15 backdoor), 65 naively malicious trajectories paired 1:1, and 20 clean ones. The question is whether an attacker who feeds malicious trajectories into an agent's skill-extraction pipeline can get a backdoor crystallized into a resident skill and routed to in later sessions. The direction matters more than the result: the skill attack surface is expanding from "what you installed" to "what your agent taught itself."
Versus skill-vetter: this one isn't security theater
In May 2026 I reviewed the first AI skill security scanner, skill-vetter, and called it security theater: grep regexes, unable to pass its own checks, with exemption rules written for itself. Side by side, the gap isn't in feature count:
| Dimension | skill-vetter (2026-05) | A.I.G skill-scan (2026-08) |
|---|---|---|
| Verdict mechanism | grep regex straight to a conclusion | regex yields leads, LLM agent yields the verdict |
| Understands code-free skills | No | Yes (SKILL.md instructions are behavior) |
| Obfuscation resistance | None | base64 tool + reversible-mojibake recovery + non-UTF-8 alerts |
| Risk taxonomy | Ad hoc | SkillTrustBench T01–T09 |
| Output format | Ad hoc | SARIF 2.1.0 (native GitHub Code Scanning) |
| Benchmark | None | SkillTrustBench, Claude Opus 4.6 F1 0.9848 |
| Reproducibility of verdicts | High (regex is deterministic) | Low (change model or temperature, change verdict) |
That last row cuts both ways: skill-vetter's determinism is worthless because its rules catch nothing, while A.I.G's nondeterminism is a real cost because its rules catch things.
The SkillTrustBench leaderboard deserves its own look:
| # | Model | F1 | Precision | Recall | FPR |
|---|---|---|---|---|---|
| 1 | Claude Opus 4.6 | 0.9848 | 0.9725 | 0.9974 | 0.0663 |
| 2 | GLM 5.1 | 0.9836 | 0.9701 | 0.9974 | 0.0723 |
| 3 | Gemini 3.5 Flash | 0.9792 | 0.9947 | 0.9641 | 0.0120 |
| 4 | Kimi 2.6 | 0.9780 | 0.9895 | 0.9667 | 0.0241 |
| 5 | DeepSeek v4 Flash | 0.9740 | 0.9868 | 0.9615 | 0.0301 |
Five models inside an 1.1% F1 band — but FPR spans 0.0120 to 0.0723, a 6x spread. Swapping models barely changes whether it finds things; it changes how often it accuses the innocent — and that is what decides whether this can gate a CI pipeline. Note that the CLI's default, deepseek-v4-flash, sits fifth on F1 while carrying less than half the leader's false-positive rate. That default is better chosen than it looks.
Eight-dimension score
| Dimension | Score | Rationale |
|---|---|---|
| Vulnerability corpus engineering | 8/10 | 2,014 clean rules with yamlcheck in CI; docked for naming inconsistency and 32.6% concentration in OpenClaw |
| Skill audit mechanism | 9/10 | The regex-as-evidence / agent-as-judge split is right; reversible-mojibake recovery is unique among peers |
| MCP audit mechanism | 6/10 | Static audit is solid, but the red team fires at an LLM-simulated target and mcp_tool isn't wired into it |
| Agent scanning | 7/10 | Detection rules as Skills makes extension nearly free; no static validation of rule quality or overlap |
| Jailbreak evaluation | 7/10 | 39 operators integrated cleanly with clear attribution; limited originality |
| Research honesty | 9/10 | Simulation labeled as simulation, CSVs published, explicit "not a general vulnerability rate" disclaimer |
| Docs-to-code fidelity | 5/10 | ".pyc bytecode bypass detection" resolves to five skip-list hits; component counts churn across the nine README languages |
| Practical adoption | 8/10 | pip install standalone + SARIF into GitHub Code Scanning; needs your own LLM key and budget |
Overall 7.4/10. Layer two and Research/ pull it up; the process that never starts and a release note with no code behind it pull it down.
Those two deductions are two faces of one problem: the self-disclosure in Research/ is scrupulous ("These values describe the controlled configuration in this release. They are not general vulnerability rates for all DeepSeek Harness deployments"), and the What's New section is not. The seam between the research team and the marketing copy is visible from outside.
Conclusion: install it, but don't treat it as a verdict
Three concrete recommendations:
- Wire
aig-skill-scaninto CI, but only block onmalicious. SARIF 2.1.0 means zero-cost integration with GitHub Code Scanning. Block merges onmalicious; annotate but don't block onsuspicious. Stay on the defaultdeepseek-v4-flash(FPR 0.0301) or use Gemini 3.5 Flash (0.0120) — the leader, Claude Opus 4.6, buys 1.1% more F1 at an FPR of 0.0663, which will burn you as a gate. Separately, add a.pyc/.pyo/.pydexistence check to the pipeline: those extensions never reach the file tree the auditor sees, so you have to see them yourself. - Trust AI Infra Scan directly; keep raw reports for the other four layers. The first is version-range comparison — reproducible and auditable. The rest are model judgments, and the "attack path" in a report needs human review before it becomes a ticket.
- Don't make go-live decisions on its MCP red team output. No real MCP process is in that loop. For actual dynamic verification, use
mcp_toolon the main scan path, or wire a real target intoredteam/target.py— that extension point is open.
If you take one thing away, make it the first line of those 14,560 runs: load_skill 15.2%, read_document 3.9%. Tools iterate and rules expire, but the structural difference — skills are loaded as instructions, documents are read as data — will not change. Every skill you install grants a stranger one instruction-level privilege. Whether A.I.G is worth installing is a matter of taste; acknowledging that first is not.
References
- Tencent Zhuque Lab — Tencent/AI-Infra-Guard on GitHub (as of 2026-08-23: 5,477 stars / 518 forks / 26 open issues)
- Tencent/AI-Infra-Guard — v4.5.2 release (2026-08-17)
- Tencent/AI-Infra-Guard — CHANGELOG.md
- Tencent/AI-Infra-Guard — skill-scan/README.md (SkillTrustBench T01–T09, SARIF 2.1.0, deduction-based scoring)
- Tencent/AI-Infra-Guard — skill_scan/utils/text_decoder.py and pre_scan.py
- Tencent/AI-Infra-Guard — mcp_scan/redteam/README.md (Crescendo / TAP; Target Runner in source-analysis mode)
- Tencent/AI-Infra-Guard — Research/deepseek-harness-security-assessment (full data and aggregation scripts for the 14,560 runs)
- Tencent/AI-Infra-Guard — Research/SkillJack (persistent skill backdoors in self-evolving agents)
- confident-ai — deepteam (upstream framework for AIG-PromptSecurity)
- DeepSeek — deepseek-harness (assessment target)
- OASIS — SARIF 2.1.0 specification
- Tencent Zhuque Lab — SkillTrustBench leaderboard (T01–T09 taxonomy and model rankings)
- Tencent Zhuque Lab — "Securing the AI Agent: A Unified Framework for Multi-Layer Agent Red Teaming" (arXiv:2606.31227)
- Tencent Zhuque Lab — "MCP Unchained: Compromising The AI Agent Ecosystem Via Its Universal Connector" (Black Hat Europe 2025)
- Zhaojiacheng Zhou — "Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems" (arXiv:2605.11891, one of the 19 papers citing A.I.G)
- Zenghao Duan et al. — "SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinement" (arXiv:2604.04989)