forcedotcom/sf-skills / plugins/builder/agentforce-adlc/skills/agentforce-test
agentforce-test
FlaggedWrite, run, and analyze structured test suites for Agentforce agents — functional AND security. TRIGGER when: user writes or modifies test spec YAML (AiEvaluationDefinition); runs sf agent test create, run, run-eval, or results commands; asks about test coverage strategy, metric selection, or custom evaluations; interprets test results or diagnoses test failures; asks about batch testing, regression suites, or CI/CD test integration; requests security testing, OWASP LLM Top 10, red-teaming, penetration testing, prompt-injection tests, a security grade, or a vulnerability assessment of an agent. DO NOT TRIGGER when: user creates, modifies, previews, or debugs .agent files (use agentforce-generate); deploys or publishes agents; writes Agent Script code; uses sf agent preview for development iteration; analyzes production session traces (use agentforce-observe); performs a static safety review of .agent file content (use agentforce-generate Section 15).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 39.3 KB |
| assets/basic-test-spec.yaml | config | 3.6 KB |
| assets/guardrail-test-spec.yaml | config | 5.8 KB |
| assets/payloads/excessive-agency.yaml | config | 4.9 KB |
| assets/payloads/misinformation.yaml | config | 6.3 KB |
| assets/payloads/output-handling.yaml | config | 4.3 KB |
| assets/payloads/prompt-injection.yaml | config | 5.5 KB |
| assets/payloads/sensitive-info-disclosure.yaml | config | 5.7 KB |
| assets/payloads/system-prompt-leakage.yaml | config | 6.3 KB |
| assets/payloads/unbounded-consumption.yaml | config | 4.9 KB |
| assets/standard-test-spec.yaml | config | 6.7 KB |
| references/action-execution.md | markdown | 5.7 KB |
| references/batch-testing.md | markdown | 12.0 KB |
| references/owasp-categories.md | markdown | 10.3 KB |
| references/preview-testing.md | markdown | 13.8 KB |
| references/remediation-guide.md | markdown | 7.6 KB |
| references/security-scoring-methodology.md | markdown | 5.1 KB |
| references/security-test-design.md | markdown | 26.4 KB |
| references/security-troubleshooting.md | markdown | 9.8 KB |
| references/test-report-format.md | markdown | 4.5 KB |
| references/troubleshooting.md | markdown | 3.7 KB |
references/test-report-format.md
View on GitHub# Test Report Format, Coverage Analysis, and CI/CD — Reference
## Summary Report
```text
Agentforce Agent Test Report
===========================================
Agent: OrderManagementAgent
Org: production
Test Cases: 6
Duration: 45.2s
Results:
Subagent Routing: 5/6 passed (83.3%)
Action Invocation: 4/6 passed (66.7%)
Grounding: 6/6 passed (100%)
Safety: 6/6 passed (100%)
Response Quality: 5/6 passed (83.3%)
Overall Score: 86.7%
Status: PASSED WITH WARNINGS
```
## Detailed Test Cases
```text
Test Case 1: "Where is my order?"
Expected Topic: order_mgmt
Actual Topic: order_mgmt (pass)
Expected Action: get_order_status
Actual Action: get_order_status (pass)
Grounding: GROUNDED (pass)
Safety Score: 0.95 (pass)
Response Quality: Relevant (pass)
Test Case 2: "I want to return this"
Expected Topic: returns
Actual Topic: order_mgmt (fail - misrouted)
Fix Applied: Expanded 'returns' subagent description
Retry Result: Correctly routed (pass)
```
## Coverage Analysis
Track which subagents and actions are tested across both modes:
| Dimension | Target | How to measure |
|-----------|--------|----------------|
| Subagent coverage | 100% of non-entry subagents | Count subagents with at least 1 test case |
| Action coverage | 100% of actions | Count actions with at least 1 test case targeting them |
| Phrasing diversity | 3+ utterances per subagent (production) | Multiple wordings per intent |
| Guardrail coverage | At least 1 off-topic test | Verify agent deflects non-relevant queries |
| Multi-turn coverage | Test subagent transitions | Conversation history tests |
| Escalation coverage | Test escalation triggers | Verify human handoff works |
## CI/CD with Testing Center
For CI/CD pipelines, use Mode B (Testing Center) for persistent regression suites:
```yaml
# .github/workflows/agent-testing.yml
name: Agent Testing
on:
pull_request:
paths:
- 'force-app/**/*.agent'
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Authenticate org
run: |
echo "${{ secrets.SFDX_AUTH_URL }}" > auth.txt
sf org login sfdx-url --sfdx-url-file auth.txt --alias testorg
- name: Deploy test suite
run: |
sf agent test create --json \
--spec tests/${{ vars.AGENT_NAME }}-testing-center.yaml \
--api-name ${{ vars.AGENT_NAME }}_CI \
--force-overwrite \
-o testorg
- name: Run tests
run: |
sf agent test run --json \
--api-name ${{ vars.AGENT_NAME }}_CI \
--wait 15 \
--result-format junit \
--output-dir test-results \
-o testorg
- name: Upload test results
uses: actions/upload-artifact@v3
with:
name: agent-test-results
path: test-results/
```
## Cross-Skill Integration (/agentforce-observe)
The /agentforce-observe skill creates test cases during its Phase 3.7 after fixing issues found through STDM session analysis. These test cases use **Testing Center format** so they can be deployed directly to the org.
### Test Case Convention
Test cases from /agentforce-observe follow Testing Center YAML format:
```yaml
# tests/<AgentApiName>-regression.yaml
name: "<AgentName> Regression Tests"
subjectType: AGENT
subjectName: <AgentApiName>
testCases:
- utterance: "find me a home in San Jose"
expectedTopic: home_search
expectedActions:
- search_homes_and_communities
- utterance: "I have a legal dispute"
expectedTopic: escalation
expectedActions:
- transfer_to_agent
```
### Deploying Cross-Skill Tests
When /agentforce-observe generates test cases, deploy them using Mode B:
```bash
# Deploy the regression test suite
sf agent test create --json \
--spec tests/<AgentApiName>-regression.yaml \
--api-name <AgentApiName>_Regression \
--force-overwrite \
-o <org>
# Run
sf agent test run --json \
--api-name <AgentApiName>_Regression \
--wait 10 \
--result-format json \
-o <org>
```
### Test File Location Convention
```text
<project-root>/
tests/
<AgentApiName>-testing-center.yaml # Full smoke suite (Mode B -- Testing Center)
<AgentApiName>-regression.yaml # Regression tests from /agentforce-observe (Mode B)
<AgentApiName>-smoke.yaml # Ad-hoc smoke tests (Mode A -- preview only)
```
Both this skill and /agentforce-observe write to the `tests/` directory using the agent's API name as prefix. Testing Center files (`-testing-center.yaml`, `-regression.yaml`) use the `name/subjectType/subjectName/testCases` format.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |