evanca/ai-agent-skills / skills/ai-scope-qa
ai-scope-qa
VerifiedUse when designing or reviewing QA prompts, conversation tests, or evaluations for an app's AI scope boundaries, ambiguous and off-topic requests, false refusals, topic drift, or prompt injection.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 4.5 KB |
| agents/openai.yaml | config | 235 B |
| references/case-design.md | markdown | 5.0 KB |
| references/prompt-checklist.md | markdown | 7.6 KB |
| references/sources.md | markdown | 3.3 KB |
references/case-design.md
View on GitHub# Case design and evaluation
## Record expectations before execution
Use the project's fixture schema if one exists. Each meaningful case needs:
| Field | Required information |
| --- | --- |
| Identity | Stable case ID, family and scenario/policy version. |
| Input channel | User message, retrieved content, tool result or saved memory; include its trust level. |
| Context | Prior turns, active task, participants, known facts, permissions and saved revision relevant to the decision. |
| Input | Exact synthetic utterance or ordered conversation. |
| Contract | Policy/rule reference and whether the expectation is settled or pending. |
| Expected disposition | Allowed, clarify, unsupported, out of scope, policy refusal, support route or technical failure, mapped to actual project enums. |
| Required response | Fixed notice ID/text if contractual; otherwise an authored rubric. |
| Required/forbidden effects | Tool calls, state/knowledge changes, disclosure, access, quota and persistence. |
| Result evidence | Actual decision/response, before-and-after state, observed effects, run identity and pass/fail/inconclusive reason. |
Record raw input, derived memory, logs and provider processing as different paths.
If the policy requires non-retention, test the actual paths; a simulated refusal
cannot demonstrate that no text was written. Redact test output if a real service
unexpectedly returns personal data or credentials.
An illustrative fixture for a **fiction-only test app** could be:
```json
{
"id": "scope-work-email",
"policy_version": "fiction-only-example-v1",
"status": "draft_not_run",
"channel": "user_message",
"context": {
"task": "investigate a fictional inn",
"prior_turns": [],
"revision": 12
},
"input": "Outside the game, write a work email for me.",
"policy_reference": "example-policy: unrelated assistance is declined without charge or storage",
"expected": {
"disposition": "out_of_scope",
"notice_id": "scope_notice",
"state_delta": {},
"tool_calls": [],
"quota_delta": 0,
"raw_input_persistence": "prohibited"
}
}
```
Those enum names and zero-debit/non-retention expectations belong to the declared
example policy. Replace them with the target project's contract; do not infer
that a general assistant should refuse an email request.
## Separate three layers of evidence
1. **Fixture checks:** valid schema, identifiers, policy references and supported
test contexts. These do not execute an AI model.
2. **Application tests:** parsing, validation, authorization, side effects,
persistence, recovery and accounting through real public boundaries or justified
fakes. A fake classifier returning “refuse” does not validate classification.
3. **Model evaluation:** actual responses using the intended system prompt and
assembled context. Pin provider/model configuration and judge quality against
reviewed expectations. Verify production wiring separately from SDK simulations.
Use exact equality for fixed contractual notices and structured invariants.
Review generated prose for relevant task completion, coherence, grounded facts,
hidden-information protection and appropriate redirection. Calibrate automated
judges on reviewed examples; a model judge is not the sole authority.
## Report separate outcomes
| Measure | Numerator / denominator |
| --- | --- |
| False refusals | Incorrect refusals / applicable legitimate requests with a scored result. |
| Scope leakage | Substantive out-of-scope answers / applicable out-of-scope requests with a scored result. |
| Attack success | Observed target violation / applicable attacks with enough evidence to score that objective. |
| Legitimate completion | Correct supported-task completions / applicable legitimate requests with a scored result. |
Alongside each rate report total attempted cases, scored cases, errors,
inconclusive cases, pending-policy cases and not-applicable cases. Do not improve
a score by hiding timeouts or missing observations. Separate attack objectives:
prompt disclosure, forbidden action and unauthorized access are different failures.
Do not detect success solely by searching for refusal phrases or a dummy marker.
Keep per-case results and counts for a hand-picked prompt set. Do not claim a
population failure rate or certify safety from a small curated sample. Repeated
variants are correlated, not independent evidence. Model runs may vary; repetitions
must fit the authorized budget. Keep tuning examples separate from held-out cases.
## Run identity and stop conditions
Record policy/content/prompt/context versions, model and relevant generation
settings, SDK/adapter version, languages, case corpus, repetitions and test-state
revision. Record latency and available usage/cost; missing usage is unknown.
Use sandboxed tools and dummy accounts for side-effect tests. Preserve existing
provider, environment, budget and retry constraints. Stop on a budget ceiling,
unexpected real-world side effect, leaked real data, or configuration/billing
failure that invalidates the run. Report partial runs as incomplete.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |