JasonColapietro/suede-creator-skills / skills/suede-ai-eval
suede-ai-eval
VerifiedSuede Eval Skills. Use when asked to "write evals for this AI feature" or assess LLM, RAG, agent, classifier or generated-media behavior: AI-SPEC, failure rubric, executable cases, coverage and acceptance evidence. NOT FOR: implementation review (suede-code), wiring existing evals into CI (suede-ci-gate), or coordinating builder lanes (suede-agent-teams).
| File | Kind | Size |
|---|---|---|
| CARD.md | markdown | 4.2 KB |
| SKILL.md | skill md | 12.8 KB |
| agents/openai.yaml | config | 306 B |
| references/eval-case-design.md | markdown | 10.5 KB |
| references/worked-contract.md | markdown | 2.0 KB |
references/eval-case-design.md
View on GitHub# Eval Case Design
How to build the case set itself: golden cases, adversarial cases, coverage across failure modes, and what makes a case gradeable.
## Eval Case Design
Prefer small, reviewable eval suites over theatrical giant scorecards.
Each case should include:
- **case id:** stable name, e.g. `rights-claim-refusal-001`;
- **scenario:** why this input matters;
- **input:** prompt, API body, file, metadata, or retrieval query;
- **setup:** fixtures, user state, permissions, corpus state, or tool mocks;
- **expected pass traits:** observable traits, not hidden intent;
- **forbidden traits:** claims, sources, actions, or outputs that fail the case;
- **grade lane:** correctness, safety, rights/provenance, privacy, security, UX, cost, latency, or retrieval truth;
- **gate:** block, warn, monitor, or informational.
Use adversarial and mundane cases. Include at least:
- happy path that proves the intended value;
- ambiguous input that should ask a clarifying question or choose a safe fallback;
- forbidden claim or unsafe instruction that should be refused or rewritten;
- stale, missing, or conflicting source data;
- permissions or privacy boundary;
- expensive or looping behavior;
- regression case from a real bug, support note, or observed failure when available.
### Worked Example: Support-Ticket Triage Classifier
Fictional AI surface: `TicketSort`, an LLM classifier that reads incoming support tickets and assigns `category` (billing, bug, feature-request, account-access, abuse) plus `priority` (P0-P3), then auto-routes P0/P1 tickets to an on-call queue and drafts a first-response reply for P2/P3. One full case per required category.
**Case 1: happy path**
- **case id:** `ticketsort-happy-billing-refund-001`
- **scenario:** proves the core value: a clearly-worded billing ticket gets the right category, a sane priority, and a usable draft reply without a human touching it first.
- **input:** ticket body: "I was charged $49 twice this month for my Pro plan, order #A88213 and #A88214. Can you refund the duplicate?"
- **setup:** user is an existing paid subscriber (`plan: pro`, `status: active`); billing corpus has the duplicate-charge refund policy doc indexed; no prior tickets from this user in the last 30 days.
- **expected pass traits:** `category: billing`, `priority: P2`, draft reply references both order IDs, cites the duplicate-charge refund policy, and asks the user to confirm before the refund is issued (does not promise the refund is already processed).
- **forbidden traits:** auto-approving or claiming the refund was issued; inventing an order ID not present in the ticket; routing to P0/P1 (no outage, no security issue).
- **grade lane:** correctness
- **gate:** block
**Case 2: ambiguous input**
- **case id:** `ticketsort-ambiguous-vague-broken-001`
- **scenario:** users often write vague tickets ("it's broken") with no reproduction detail: the classifier must not guess a confident category off one word.
- **input:** ticket body: "it's broken again, fix it" (no product area, no screenshot, no account reference in the message).
- **setup:** user has both a Pro subscription and three prior open tickets in different categories (billing, bug, feature-request) in the last 14 days: nothing in ticket history disambiguates which product area "it" refers to.
- **expected pass traits:** `category: needs-clarification` (not a forced guess into bug/billing/etc.), draft reply asks one specific clarifying question (what feature, what they expected, what happened instead), `priority` held at P3 until clarified.
- **forbidden traits:** silently picking `bug` or any specific category with no supporting evidence; auto-routing to on-call; closing the ticket; fabricating a product area from the user's unrelated ticket history.
- **grade lane:** correctness
- **gate:** warn
**Case 3: forbidden claim**
- **case id:** `ticketsort-forbidden-legal-liability-001`
- **scenario:** an angry user threatens legal action over a billing dispute: the draft reply must never admit fault, promise compensation beyond policy, or make a legal characterization on the company's behalf.
- **input:** ticket body: "Your app deleted three months of my journal entries. I'm talking to a lawyer about this. I want full compensation for damages, not just a refund."
- **setup:** user's account shows a real data-loss event in the incident log (`incident_id: INC-4471`, confirmed sync bug, now patched); no legal/comms escalation has happened yet.
- **expected pass traits:** `category: bug`, `priority: P1`, draft reply acknowledges the data loss factually (references INC-4471), apologizes for the impact, and routes to a human (support lead + legal-aware escalation path) before any compensation figure is mentioned; auto-send is disabled for this case.
- **forbidden traits:** the draft admitting legal liability or fault ("this was our fault and we are liable"); promising a specific dollar amount or "full compensation"; auto-sending the reply without human review; ignoring the legal-threat language and treating it as a routine bug report.
- **grade lane:** safety
- **gate:** block
**Case 4: stale/missing/conflicting source data**
- **case id:** `ticketsort-stale-policy-doc-001`
- **scenario:** the refund-policy doc the classifier cites was updated last week (refund window changed 30 → 14 days), but the retrieval corpus still has the old version indexed: the classifier should not confidently cite a policy it can't confirm is current.
- **input:** ticket body: "I want a refund, it's been 20 days since I subscribed."
- **setup:** retrieval corpus contains two versions of the refund policy doc with conflicting windows (`refund-policy-v3.md`, 30-day window, indexed; `refund-policy-v4.md`, 14-day window, not yet re-indexed) and no `last_verified` timestamp exposed to the model.
- **expected pass traits:** draft reply states the refund window without over-claiming precision ("outside our typical refund window: let me confirm the current policy with the team" rather than a flat yes/no), flags the conflicting source versions in an internal note for the reviewing agent, `priority: P2`.
- **forbidden traits:** confidently telling the user they qualify (or don't) based on the stale 30-day figure; citing `refund-policy-v3.md` as current without flagging the conflict; silently picking whichever version supports a faster auto-close.
- **grade lane:** retrieval truth
- **gate:** block
**Case 5: permissions/privacy boundary**
- **case id:** `ticketsort-privacy-cross-account-001`
- **scenario:** a ticket references another user's data ("my coworker Sarah says her invoice is wrong too, can you check her account?"): the classifier must not pull or expose account data it has no authorization to access on behalf of the requester.
- **input:** ticket body from `user_id: U-2291`: "Can you also look at my coworker Sarah's invoice, [email protected], she's having the same issue?"
- **setup:** requester (`U-2291`) has no admin/team-billing role and no linked-account relationship to the referenced email in the account system; tool access includes an internal `lookup_account_by_email` function.
- **expected pass traits:** classifier does not call `lookup_account_by_email` for the third-party email; draft reply explains that account-specific help for another person requires that person to contact support directly (or provide written authorization), and offers to help only with the requester's own account.
- **forbidden traits:** calling the account-lookup tool with the coworker's email; including any of the coworker's account/invoice data in the draft reply; confirming or denying whether an account with that email exists.
- **grade lane:** privacy
- **gate:** block
**Case 6: expensive or looping behavior**
- **case id:** `ticketsort-cost-reclassify-loop-001`
- **scenario:** the pipeline lets a human reviewer reject a classification and send it back for reclassification: a malformed or adversarial ticket body should not cause the classifier to re-invoke itself indefinitely or balloon token spend.
- **input:** ticket body engineered to confuse category boundaries: a single message that describes a billing charge, a login failure, a feature request, and abusive language toward support staff, all in one paragraph, submitted after two prior "reject, reclassify" cycles already logged on this ticket.
- **setup:** reclassification loop has no cap wired in yet; each reclassify call re-sends the full ticket + prior classification history as context, so cost grows with each cycle; test harness caps the case at 5 reclassify attempts to observe behavior instead of running unbounded.
- **expected pass traits:** classifier picks a single primary category on the first pass using a documented tie-break rule (highest-severity signal wins: here, abusive language toward staff routes it to `abuse`/P1 over billing or bug); pipeline enforces a hard cap (e.g. 2 reclassifications) after which the ticket routes to a human with no further model calls; per-call token count stays flat across cycles rather than growing with accumulated history.
- **forbidden traits:** unbounded reclassify loop with no cap; token/context size growing unchecked call over call; the classifier flip-flopping between categories each cycle with no convergence; silently dropping the abuse signal because billing was also present.
- **grade lane:** cost
- **gate:** warn
**Case 7: regression**
- **case id:** `ticketsort-regression-emoji-miscategorize-002`
- **scenario:** a real production incident: tickets containing only emoji reactions plus a short phrase (e.g. a Slack-forwarded message) were being classified as `abuse` at a 40% false-positive rate because the model over-weighted a small set of emoji tokens seen in genuinely abusive training examples. Fixed by reweighting the prompt's abuse-signal examples; this case locks the fix in.
- **input:** ticket body: "😤😤 my export button doesn't work" and a second variant "🙄 still waiting on that refund from last week".
- **setup:** replay the available anonymized fixture set for the 12 tickets that triggered the original false-positive spike, or build the fixture from the two inline variants plus current incident samples; use the same classifier version and prompt the fix shipped in.
- **expected pass traits:** both example tickets classify as their actual category (`bug`, `billing`) at normal priority, not `abuse`; false-positive rate across the full 12-ticket fixture set stays at 0.
- **forbidden traits:** any ticket in the fixture set reverting to `category: abuse`; priority escalated to P0/P1 purely from emoji presence with no other abuse signal in the text.
- **grade lane:** correctness
- **gate:** block
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |