Skip to content

yeaight7/agent-powerups / plugins/agent-evaluation-lab/skills/prompt-evaluation-runner

prompt-evaluation-runner

Verified

Use when evaluating prompts, LLM outputs, red-team suites, or model behavior with local eval configs and safe provider/cost controls.

FileKindSize
SKILL.mdskill md3.3 KB
references/eval-config-patterns.mdmarkdown1.3 KB

references/eval-config-patterns.md

View on GitHub
# Eval Config Patterns

## Minimal Shape

```yaml
description: "Short behavior check"

prompts:
  - file://prompts/main.txt

providers:
  - id: openai:chat:gpt-5-mini
    label: gpt-5-mini
    config:
      temperature: 0

defaultTest:
  assert:
    - type: cost
      threshold: 0.01
    - type: latency
      threshold: 5000

tests:
  - vars:
      input: "case text"
    assert:
      - type: contains
        value: "expected phrase"
```

## Assertion Order

Use deterministic assertions first:

| Need | Assertion |
| --- | --- |
| exact output | `equals` |
| required phrase | `contains`, `icontains`, `contains-all` |
| forbidden text | `not-contains`, `not-regex` |
| structured output | `is-json`, JSON schema |
| cheap guardrail | `cost`, `latency`, `word-count` |
| custom local logic | `javascript`, `python` |
| fuzzy quality | `similar`, `rouge-n`, `bleu` |
| subjective quality | `llm-rubric` with pinned grader |

## Env Rules

- Use `{{env.NAME}}`, not shell `$NAME`.
- Do not hardcode secrets.
- Do not ask user to paste keys into chat.
- State provider, model, expected cost, and network target before running.

## Analysis Template

```text
Eval:
Provider/model:
Cases run:
Pass/fail:
Deterministic failures:
Model-graded failures:
Likely causes:
Next smallest fix:
```
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall