yeaight7/agent-powerups / plugins/agent-evaluation-lab/skills/prompt-evaluation-runner
prompt-evaluation-runner
VerifiedUse when evaluating prompts, LLM outputs, red-team suites, or model behavior with local eval configs and safe provider/cost controls.
Prompt Evaluation Runner
When to use
Use when you need to evaluate an LLM app, test a prompt systematically, or run red-team/vulnerability scans against a target model or application.
Requirements / Checks
- Check if an evaluation tool is defined in project deps, scripts, lockfiles, or local toolchain (e.g.,
promptfoo,evals,braintrust). - Do not run unvetted remote runners without checking the project's toolchain first (e.g., avoid
npx promptfoo@latestifpromptfoois already installed locally). - If no runner exists, ask before adding a dev dependency or using an ephemeral runner.
- Confirm expected cost, provider, API keys, and network target before any execution.
Workflow
-
Define risk — state target behavior, failure mode, provider(s), and budget limits before writing any config.
-
Choose assertions — prefer deterministic checks first:
Assertion type When to use contains/not-containsOutput must include/exclude specific text regexStructured output pattern (e.g., JSON key present) json-schemaOutput must conform to a schema costMust stay under a token/dollar budget latencyMust respond within N ms javascript/pythonCustom logic when simpler types don't fit Model grader Last resort — only for subjective quality checks -
Use model graders sparingly — pin the grader model and provider explicitly; document the cost and non-determinism risk.
-
Minimal config structure:
description: "Test that the summarizer stays under 200 words" providers: - id: openai:gpt-4o-mini config: temperature: 0 prompts: - "Summarize: {{input}}" defaultTest: assert: - type: javascript value: output.split(' ').length < 200 tests: - vars: input: "{{env.TEST_DOCUMENT}}" -
Handle env safely — use
{{env.VAR_NAME}}for all secrets and inputs. Never hardcode API keys or sensitive data in config files. -
Execute locally — run the smallest suite first. Ask before running long, paid, red-team, or production-targeted suites.
-
Analyze failures — classify before fixing:
- Prompt failure (model output is wrong)
- Provider variance (non-deterministic model)
- Flaky grader (model grader is inconsistent)
- Bad fixture (test input is unrealistic)
- Config mistake (assertion logic error)
Safety Constraints
- Do NOT log, echo, or store API keys in configuration files or chat output.
- Do NOT run evaluations against production endpoints without user consent.
- Do not execute arbitrary remote code or unvetted plugins during evaluation.
Validation / Done Criteria
- Eval config is valid, minimal, and uses
{{env.VAR}}references for secrets. - Deterministic assertions exist where possible; model grader use is documented and justified.
- Run scope, provider, and estimated cost are reported before execution.
- Results are summarized without leaking sensitive input data.
References
references/eval-config-patterns.md
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 468k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 321k | Updated | Claude CodeCodex | Verified | Install |