mikeparcewski/wicked-garden / skills/agentic-safety-reviewer
wicked-garden-agentic-safety-reviewer
UnverifiedGuardrails, prompt injection defense, PII protection, human-in-the-loop gates, and hallucination mitigation for agentic systems. Use when: safety review of an AI agent system, guardrail assessment, prompt injection audit, PII/compliance exposure check, HITL gate verification, or as a parallel worker in a heavyweight wicked-garden-agentic review. Cross-ref: EXECUTING injection/jailbreak/refusal-rate/drift probes against a live LLM feature with evidence artifacts + a ledger verdict is `wicked-garden-qe-ai-feature-test-engineer` (executor); THIS skill is the design-time safety review.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 6.1 KB |
| refs/output-format.md | markdown | 7.4 KB |
| refs/process.md | markdown | 8.9 KB |
refs/output-format.md
View on GitHub# Safety review — the report template
Loaded on demand by the `wicked-garden-agentic-safety-reviewer` worker skill. Emit the template verbatim with the placeholders filled.
## Output Format
````markdown
## Safety Review: {Project Name}
**Review Date**: {date}
**Risk Level**: {CRITICAL/HIGH/MEDIUM/LOW}
**Codebase Path**: {path}
### Executive Summary
{2-3 sentence summary of safety posture and critical risks}
### Risk Profile
| Category | Findings | Critical | High | Medium | Low |
|----------|----------|----------|------|--------|-----|
| Prompt Injection | {count} | {count} | {count} | {count} | {count} |
| PII Protection | {count} | {count} | {count} | {count} | {count} |
| Guardrails | {count} | {count} | {count} | {count} | {count} |
| Human-in-the-Loop | {count} | {count} | {count} | {count} | {count} |
| Hallucination Risk | {count} | {count} | {count} | {count} | {count} |
### Prompt Injection Assessment
**Status**: {PROTECTED/VULNERABLE/CRITICAL}
**Direct Injection**:
- [ ] User input is not directly concatenated into prompts
- [ ] Clear delimiters separate system/user content
- [ ] Instruction hierarchy is enforced
- [ ] Injection patterns are detected and blocked
**Findings**:
- **CRITICAL**: {file:line} - User input directly in prompt without validation
```python
prompt = f"You are a helper. {user_input}" # VULNERABLE
```
**Fix**: Use structured prompts with clear boundaries
- **HIGH**: {file:line} - No injection pattern detection
**Indirect Injection**:
- [ ] External content is sanitized before inclusion
- [ ] RAG documents are from trusted sources
- [ ] Web scraping results are validated
- [ ] API responses are filtered
**Findings**:
- **HIGH**: {file:line} - Untrusted URL content included in prompt
- **MEDIUM**: {file:line} - Search results without sanitization
**Recommendations**:
1. Implement input validation with injection pattern detection
2. Add clear delimiters: "User Query:", "Instructions:", etc.
3. Sanitize all external content before prompt inclusion
### PII Protection Assessment
**Status**: {COMPLIANT/PARTIAL/NON_COMPLIANT}
**Detection**:
- [ ] PII patterns detected in inputs
- [ ] PII patterns detected in outputs
- [ ] PII patterns detected in logs
**Findings**:
- **CRITICAL**: {file:line} - Email addresses logged in plaintext
- **HIGH**: {file:line} - SSN patterns not redacted in responses
**Redaction**:
- [ ] Input PII is redacted before storage
- [ ] Output PII is redacted before returning
- [ ] Log PII is masked or hashed
**Findings**:
- **HIGH**: {file:line} - No redaction mechanism implemented
- **MEDIUM**: {file:line} - Inconsistent redaction across agents
**Compliance**:
- [ ] GDPR: Right to erasure implemented
- [ ] CCPA: Do not sell enforcement
- [ ] HIPAA: PHI encryption at rest and in transit (if applicable)
**Findings**:
- **HIGH**: No data retention policy for PII
- **MEDIUM**: PII stored without encryption
**Recommendations**:
1. Implement PII detection library (e.g., regex + ML-based)
2. Add redaction layer for all inputs/outputs/logs
3. Create PII cleanup policy and scheduled jobs
4. Encrypt PII storage and transmission
### Guardrails Assessment
**Status**: {IMPLEMENTED/PARTIAL/MISSING}
**Input Guardrails**: {PRESENT/MISSING}
- [ ] Length limits enforced
- [ ] Content validation exists
- [ ] Injection detection active
- [ ] Rate limiting per user/session
- [ ] Toxicity filtering enabled
**Findings**:
- **CRITICAL**: No input validation at agent entry points
- **HIGH**: No rate limiting - DoS risk
- **MEDIUM**: No toxicity filtering
**Output Guardrails**: {PRESENT/MISSING}
- [ ] PII redaction in responses
- [ ] Toxicity filtering in generated content
- [ ] Citation requirements enforced
- [ ] Uncertainty disclaimers added
- [ ] Content policy compliance
**Findings**:
- **HIGH**: No output validation before returning
- **MEDIUM**: No citation requirements
**Action Guardrails**: {PRESENT/MISSING}
- [ ] Critical actions require confirmation
- [ ] Audit logging for all actions
- [ ] Approval workflow for destructive actions
- [ ] Timeout and fallback strategies
**Findings**:
- **CRITICAL**: Delete operations not gated
- **HIGH**: Payment actions lack human approval
- **MEDIUM**: No audit trail for actions
**Recommendations**:
1. Implement three-layer guardrails: input, output, action
2. Add rate limiting with per-user quotas
3. Create approval workflow for critical actions
4. Enable comprehensive audit logging
### Human-in-the-Loop Assessment
**Status**: {IMPLEMENTED/PARTIAL/MISSING}
**Escalation Strategy**: {CLEAR/UNCLEAR/MISSING}
- [ ] Escalation triggers are documented
- [ ] Low-confidence threshold defined
- [ ] High-stakes domains identified
- [ ] Critical actions flagged
**Findings**:
- **HIGH**: No escalation logic for low-confidence scenarios
- **MEDIUM**: High-stakes domains not identified
**Review Workflow**: {IMPLEMENTED/MISSING}
- [ ] Reviewer assignment logic exists
- [ ] Timeout strategy defined
- [ ] Fallback for no reviewer available
- [ ] Decision tracking implemented
**Findings**:
- **HIGH**: No timeout strategy - can block indefinitely
- **MEDIUM**: Reviewer decisions not tracked
**Recommendations**:
1. Define confidence threshold for escalation (e.g., < 0.7)
2. Identify high-stakes domains: medical, legal, financial
3. Implement timeout with safe fallback (default: deny)
4. Add decision tracking for feedback loop
### Hallucination Mitigation Assessment
**Status**: {STRONG/MODERATE/WEAK}
**Grounding Mechanisms**: {PRESENT/MISSING}
- [ ] Citations required for factual claims
- [ ] RAG retrieval before answering
- [ ] Tool use preferred over memorization
- [ ] Verification against knowledge base
- [ ] Confidence scoring enabled
**Findings**:
- **MEDIUM**: No citation requirements - hallucination risk
- **MEDIUM**: Confidence scores not computed
- **LOW**: "I don't know" responses not encouraged
**Detection**: {ACTIVE/PASSIVE/MISSING}
- [ ] Cross-checking factual claims
- [ ] Multi-agent verification for critical info
- [ ] Contradiction detection
- [ ] Source availability check
**Findings**:
- **HIGH**: No fact verification mechanism
- **MEDIUM**: Single-agent responses without verification
**Recommendations**:
1. Require citations for all factual claims
2. Implement confidence scoring and return to user
3. Add multi-agent verification for high-stakes answers
4. Encourage "I don't know" over guessing
### Critical Vulnerabilities
**Priority 1 (Fix Immediately)**:
1. {vulnerability} - {location}
- **Risk**: {description}
- **Fix**: {specific action}
- **Effort**: {LOW/MEDIUM/HIGH}
**Priority 2 (Fix Before Production)**:
1. {vulnerability} - {location}
- **Risk**: {description}
- **Fix**: {specific action}
### Secure Patterns Observed
- {positive finding}
- {positive finding}
### Next Steps
1. **Immediate**: {critical fix}
2. **Short-term**: {high priority fix}
3. **Medium-term**: {improvement}
4. **Long-term**: {strategic enhancement}
### Cross-Skill Coordination
**Defer to**:
- **wicked-garden-agentic-architect**: For Layer 5 architecture validation
- **wicked-garden-agentic-performance-analyst**: For rate limiting and throttling implementation
- **frameworks knowledge skill** (`skills/agentic/frameworks/`): For framework-native safety features
**Collaborate with**:
- The architect skill on guardrail placement in the five-layer architecture (see the agentic-patterns knowledge module)
- The performance-analyst skill on efficient validation strategies
````
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |