Skip to content

jaktestowac/awesome-copilot-for-testers / plugins/testing-llm-guardrails/skills/testing-llm-guardrails

testing-llm-guardrails

Flagged

Tests the runtime validation around model output - schema conformance, PII leakage, moderation, refusal behaviour, tool-call safety - and the adversarial suite that tries to defeat it: direct prompt injection, jailbreaks, and indirect injection through retrieved documents, tool results and user content. Use when an LLM feature reaches real users, when a model can call tools or read untrusted content, when an AI feature needs a security-style test suite, or when asked whether a chatbot or agent can be manipulated.

Security report

Flagged

Static analysis of one exact version. It reduces risk and shows evidence; it does not certify that a skill is safe.

Commit
8910672e674853caf0e68dbffc59c39869990203
Content hash
86fd98d4a29f8342654c7e47cde0d019c840fea332d3305b73b0caaf2cdf4655
Scanned
2026-11-17
Scanner version
skillxone-scan-1.0.0
Policy version
0.1-draft
Decision
1 open critical or high finding.
Verification expires
Not applicable

Findings (1 high, 2 low, 1 info)

  • highPI-IGNORE-INSTRUCTIONSresources/injection-test-corpus.md:26

    The text tells the agent to ignore or override its existing instructions, a common prompt-injection pattern.

    Shapes: direct override of prior instructions; a claim of higher authority ("system message update:", "administrator override"); a claim that the task has changed; instructions embedded in what looks like data; a fake conversation boundary…

    Suggested fix: Remove the instruction. A skill should add guidance for its task, not cancel the instructions the agent already follows.

    Report as false positive

  • lowPI-REVEAL-PROMPTSKILL.md:43

    The text asks the agent to disclose its system prompt or hidden instructions. Severity lowered from high because the surrounding text warns against it.

    - **Prompt instructions are not enforcement.** "Never reveal the system prompt" is a hope. A regex on the output is a check. Authorisation in the tool handler is a control.

    Suggested fix: Remove the request. A skill has no need to read or reveal the system prompt.

    Report as false positive

  • lowPI-IGNORE-INSTRUCTIONSSKILL.md:49

    The text tells the agent to ignore or override its existing instructions, a common prompt-injection pattern. Severity lowered from high because it appears in a quotation.

    - **A corpus, not a handful of tricks.** "Ignore previous instructions" is one case. Categories, versioned and grown from real attempts, is a suite.

    Suggested fix: Remove the instruction. A skill should add guidance for its task, not cancel the instructions the agent already follows.

    Report as false positive

  • infoPI-JAILBREAK-PERSONASKILL.md:46

    The text uses language typical of jailbreak prompts. Severity lowered from medium because the surrounding text warns against it.

    - **Indirect injection is the realistic attack.** Nobody needs to type a jailbreak if a retrieved document can carry one. Any feature with retrieval or tool output has this surface.

    Suggested fix: Remove the jailbreak framing.

    Report as false positive

Checks run

  • prompt-injection
  • dangerous-commands
  • credentials
  • network
  • file-access
  • integrity

Checks skipped

  • dependencies: No dependency manifest in the skill directory.
  • gitleaks: gitleaks is not installed on this host.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall