existential-birds/beagle / plugins/beagle-testing/skills/gen-test-plan
gen-test-plan
VerifiedAnalyze repo, detect stack, trace changes to user-facing entry points, generate E2E YAML test plan
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 17.0 KB |
| references/stack-discovery.md | markdown | 6.4 KB |
| references/test-case-generation.md | markdown | 6.7 KB |
references/test-case-generation.md
View on GitHub# Test Case Generation
This reference keeps the long test-template examples and prioritization guidance out of `SKILL.md`.
## Critical: E2E Only
Every test case must exercise the **real built artifact** — the actual binary, server, or UI — exactly as a human QA tester would. A test case that invokes the project's automated test runner (`cargo test`, `pytest`, `npm test`, `go test`, `mix test`, etc.) is **never valid**. Those tests already run in CI. The purpose of this plan is to verify behavior that automated tests cannot: real end-to-end workflows through the actual user interface (CLI, HTTP, browser).
## Step 5: Generate Test Cases
Before generating test cases, answer: "What does this change do for the end user?"
Then ask: "How would a human tester — who has never seen the code — verify this works?"
Generate tests in this order:
1. Core functionality tests first - exercise the primary behavioral change through the actual user-facing interface.
2. Configuration/admin tests second - support the feature but do not replace the core test.
### CLI Applications (shell tests)
```yaml
- id: TC-XX
name: <Describe what user action this represents>
context: |
<Which files changed and why this subcommand is affected>
steps:
# Build the binary first (or reference it from setup.build)
- run: <invoke the CLI binary with real arguments>
- run: <verify the effect — check stdout, query database, inspect files>
expected: |
<Exit code, stdout content, side effects (files created, DB rows, etc.)>
```
**CLI test examples by scenario:**
```yaml
# Subcommand that writes to a database
- id: TC-01
name: "volant plan creates a workflow in PostgreSQL"
steps:
- run: ./target/debug/volant plan "Fix login bug" --description "Users can't log in" --sandbox local
- run: psql "${DATABASE_URL}" -c "SELECT id, state FROM workflows ORDER BY created_at DESC LIMIT 1"
expected: |
Exit code 0. A new workflow row exists with state 'pending' or further.
# Subcommand that outputs to stdout
- id: TC-02
name: "volant status lists workflows"
steps:
- run: ./target/debug/volant status --all
expected: |
Exit code 0. Outputs a table or list of workflows. Does not crash on empty database.
# Subcommand with --dry-run
- id: TC-03
name: "volant run --dry-run validates without executing"
steps:
- run: ./target/debug/volant run example.toml --dry-run
expected: |
Exit code 0. Prints the resolved workflow structure. Does not execute any nodes.
# Error handling — missing config
- id: TC-04
name: "volant plan errors gracefully without API key"
steps:
- run: env -u ANTHROPIC_API_KEY ./target/debug/volant plan "test" 2>&1 || true
expected: |
Non-zero exit code. Stderr contains a meaningful error about missing configuration,
not a panic or stack trace.
```
### API Endpoints (curl tests)
```yaml
- id: TC-XX
name: <Describe what user action this represents>
context: |
<Which files changed and why this endpoint is affected>
steps:
- action: curl
method: <GET|POST|PUT|DELETE>
url: http://localhost:<port>/<path>
headers:
Content-Type: application/json
body: <JSON body if needed>
expected: |
<HTTP status code, response body shape, side effects>
```
### Database Verification
```yaml
- id: TC-XX
name: <Describe what data change this verifies>
context: |
<Which migration or data-writing code changed>
steps:
- run: <command that triggers the data write>
- run: psql "${DATABASE_URL}" -c "<SQL query to verify>"
expected: |
<Expected rows, column values, or schema state>
```
**Database test examples:**
```yaml
# Migration applies cleanly
- id: TC-05
name: "Session tables migration creates expected schema"
steps:
- run: psql "${DATABASE_URL}" -c "\dt sessions"
- run: psql "${DATABASE_URL}" -c "\d sessions"
expected: |
The 'sessions' table exists with columns matching the migration definition.
# Data roundtrip through the application
- id: TC-06
name: "Checkpoint save and resume preserves workflow state"
steps:
- run: ./target/debug/volant plan "test checkpoint" --sandbox local
- run: psql "${DATABASE_URL}" -c "SELECT workflow_id, state FROM checkpoints ORDER BY created_at DESC LIMIT 1"
expected: |
A checkpoint row exists for the workflow with the expected state payload.
```
### UI Routes (agent-browser CLI tests)
```yaml
- id: TC-XX
name: <Describe the user journey>
context: |
<Which files changed and why this route is affected>
steps:
- run: agent-browser open http://localhost:<port>/<path>
- run: agent-browser snapshot -i
note: Capture interactive elements with refs
- run: agent-browser fill @<ref> "<test value>"
- run: agent-browser click @<ref>
- run: agent-browser wait --url "**/<expected-path>"
- run: agent-browser snapshot -i
note: Verify final state
- run: agent-browser screenshot docs/testing/evidence/tc-XX.png
expected: |
<Natural language description of expected behavior>
evidence:
screenshot: docs/testing/evidence/tc-XX.png
```
### Process Lifecycle (TUI / long-running)
```yaml
- id: TC-XX
name: <Describe the process behavior>
steps:
# Start the process in background, give it time to initialize, then verify
- run: timeout 5 ./target/debug/volant 2>&1 || true
- run: <verify it started correctly — check stderr output, temp files, etc.>
expected: |
Process starts without crash. Produces expected initial output.
Exits cleanly on timeout/interrupt.
```
### Test Case Guidelines
- **Never invoke the project's test runner** — every step must be a real user action
- At least one test per affected user-facing entry point
- CLI tests for command-line tools — invoke the binary directly
- curl tests for HTTP servers
- Browser tests for web UIs — always use real `agent-browser` CLI commands
- Database tests when migrations or data-writing code changed
- Include authentication/config steps if commands require credentials
- Test error paths — missing config, bad input, unreachable services
- Always snapshot before interacting and re-snapshot after navigation or DOM changes
## Step 6: Write YAML Test Plan
Create `docs/testing/test-plan.yaml` with metadata, setup, prerequisites, and the generated tests.
## Step 7: Report Summary
Report the generated file, detected stack, tests generated, entry-point coverage, and next steps.
## Step 8: Verification
Verify the YAML file exists, parses successfully, and includes the required top-level keys.
**Additional verification:** Grep every `run:` step for test runner commands (`cargo test`, `pytest`, `npm test`, `go test`, `mix test`, `jest`, `vitest`). If any are found, the plan fails — replace with real E2E actions.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |