yeaight7/agent-powerups / plugins/agent-evaluation-lab/skills/prompt-evaluation-runner
prompt-evaluation-runner
VerifiedUse when evaluating prompts, LLM outputs, red-team suites, or model behavior with local eval configs and safe provider/cost controls.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 3.3 KB |
| references/eval-config-patterns.md | markdown | 1.3 KB |
references/eval-config-patterns.md
View on GitHub# Eval Config Patterns
## Minimal Shape
```yaml
description: "Short behavior check"
prompts:
- file://prompts/main.txt
providers:
- id: openai:chat:gpt-5-mini
label: gpt-5-mini
config:
temperature: 0
defaultTest:
assert:
- type: cost
threshold: 0.01
- type: latency
threshold: 5000
tests:
- vars:
input: "case text"
assert:
- type: contains
value: "expected phrase"
```
## Assertion Order
Use deterministic assertions first:
| Need | Assertion |
| --- | --- |
| exact output | `equals` |
| required phrase | `contains`, `icontains`, `contains-all` |
| forbidden text | `not-contains`, `not-regex` |
| structured output | `is-json`, JSON schema |
| cheap guardrail | `cost`, `latency`, `word-count` |
| custom local logic | `javascript`, `python` |
| fuzzy quality | `similar`, `rouge-n`, `bleu` |
| subjective quality | `llm-rubric` with pinned grader |
## Env Rules
- Use `{{env.NAME}}`, not shell `$NAME`.
- Do not hardcode secrets.
- Do not ask user to paste keys into chat.
- State provider, model, expected cost, and network target before running.
## Analysis Template
```text
Eval:
Provider/model:
Cases run:
Pass/fail:
Deterministic failures:
Model-graded failures:
Likely causes:
Next smallest fix:
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |