forcedotcom/sf-skills / plugins/builder/agentforce-adlc/skills/agentforce-test
agentforce-test
FlaggedWrite, run, and analyze structured test suites for Agentforce agents — functional AND security. TRIGGER when: user writes or modifies test spec YAML (AiEvaluationDefinition); runs sf agent test create, run, run-eval, or results commands; asks about test coverage strategy, metric selection, or custom evaluations; interprets test results or diagnoses test failures; asks about batch testing, regression suites, or CI/CD test integration; requests security testing, OWASP LLM Top 10, red-teaming, penetration testing, prompt-injection tests, a security grade, or a vulnerability assessment of an agent. DO NOT TRIGGER when: user creates, modifies, previews, or debugs .agent files (use agentforce-generate); deploys or publishes agents; writes Agent Script code; uses sf agent preview for development iteration; analyzes production session traces (use agentforce-observe); performs a static safety review of .agent file content (use agentforce-generate Section 15).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 39.3 KB |
| assets/basic-test-spec.yaml | config | 3.6 KB |
| assets/guardrail-test-spec.yaml | config | 5.8 KB |
| assets/payloads/excessive-agency.yaml | config | 4.9 KB |
| assets/payloads/misinformation.yaml | config | 6.3 KB |
| assets/payloads/output-handling.yaml | config | 4.3 KB |
| assets/payloads/prompt-injection.yaml | config | 5.5 KB |
| assets/payloads/sensitive-info-disclosure.yaml | config | 5.7 KB |
| assets/payloads/system-prompt-leakage.yaml | config | 6.3 KB |
| assets/payloads/unbounded-consumption.yaml | config | 4.9 KB |
| assets/standard-test-spec.yaml | config | 6.7 KB |
| references/action-execution.md | markdown | 5.7 KB |
| references/batch-testing.md | markdown | 12.0 KB |
| references/owasp-categories.md | markdown | 10.3 KB |
| references/preview-testing.md | markdown | 13.8 KB |
| references/remediation-guide.md | markdown | 7.6 KB |
| references/security-scoring-methodology.md | markdown | 5.1 KB |
| references/security-test-design.md | markdown | 26.4 KB |
| references/security-troubleshooting.md | markdown | 9.8 KB |
| references/test-report-format.md | markdown | 4.5 KB |
| references/troubleshooting.md | markdown | 3.7 KB |
references/troubleshooting.md
View on GitHub# Troubleshooting, Best Practices, and Dependencies — Reference
## Common Issues
| Issue | Cause | Solution |
|-------|-------|----------|
| Session timeout | Long-running tests | Split into smaller batches |
| Trace not found | CLI version issue | Update to sf CLI 2.131.0+ |
| `Nonexistent flag: --simulate-actions` on `preview start` | CLI older than 2.131.0 (plugin-agent < 1.32.16) | Update the CLI; below 2.131.0 only `--use-live-actions` exists |
| `Nonexistent flag: --no-prompt` on `preview end` | CLI older than 2.135.5 | Drop the flag — it only affects `end --all`, and single-session `end` never prompts |
| Action mock fails | Complex inputs | Use `--use-live-actions` flag |
| Context variables missing | Preview limitation | Use Runtime API for context tests |
| `jq` parse error on preview output | Control characters in CLI output | Use Python `re.sub` + `json.loads` (see below). `tr` via bash pipes is unreliable -- control chars survive `echo "$VAR"` expansion. |
### Defensive JSON Parsing
`sf agent preview` output may contain control characters (e.g. `\x08`, `\x1b`) that break `jq` and `json.loads`. Always sanitize before parsing.
**Use Python `re.sub`** -- this is the only reliable approach. The `tr` command via `echo "$VAR" | tr -d ...` is unreliable because bash variable expansion and `echo` can re-introduce or mangle control characters:
```bash
# Recommended: Python re.sub (handles all control characters reliably)
python3 -c "
import json, sys, re
raw = sys.stdin.read()
clean = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f]', '', raw)
data = json.loads(clean)
print(json.dumps(data.get('result', {}), indent=2))
" <<< "$RESPONSE"
```
## Debug Mode
Enable detailed logging for preview sessions:
```bash
# Enable SF CLI debug output
export SF_LOG_LEVEL=debug
# Run preview with verbose output (--authoring-bundle for local traces)
sf agent preview start --authoring-bundle MyAgent --simulate-actions -o myorg --json 2>&1 | tee /tmp/preview_debug.json
```
### `MissingModeFlag` on `preview start`
`When using --authoring-bundle, you must specify either --use-live-actions or --simulate-actions.` Add `--simulate-actions` (or `--use-live-actions` if you want real Apex/Flow execution) to `start`. Do **not** add it to `send` or `end` — they reject it with `Nonexistent flag`.
### `RequiresProjectError` on any `preview` subcommand
`This command is required to run from within a Salesforce project directory.` Run the command from the directory containing `sfdx-project.json`.
## Best Practices
### Test Strategy
1. **Start with smoke tests** - Basic happy path scenarios
2. **Add edge cases** - Boundary conditions, invalid inputs
3. **Test transitions** - Multi-turn conversations
4. **Verify guardrails** - Off-topic and safety boundaries
5. **Performance baseline** - Establish acceptable response times
### Test Maintenance
- Version test cases with agent versions
- Update expected outputs when agent evolves
- Archive historical test results
- Monitor test flakiness and address root causes
## Dependencies
This skill uses `sf` CLI commands directly. Required tools:
- `sf` CLI **2.131.0+** (plugin-agent 1.32.16+) — the floor for `preview start --authoring-bundle --simulate-actions`, which every example here uses. `start`/`send`/`end` as separate subcommands need 1.28.0; `--simulate-actions` needs 1.32.16 = CLI 2.131.0.
- `jq` (system) - JSON processing
- `python3` - For result parsing scripts
## Exit Codes
| Code | Meaning | Description |
|------|---------|-------------|
| 0 | All tests passed | Safe to deploy |
| 1 | Some tests failed | Review failures before deploying |
| 2 | Critical test failure | Block deployment |
| 3 | Test execution error | Fix test infrastructure |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |