Skip to content

urmzd/dotfiles / dot_agents/skills/create-oss-skill

create-oss-skill

Verified

Create well-formed Agent Skills following the agentskills.io specification. Scaffold directories, write SKILL.md files, bundle scripts, and structure instructions for progressive disclosure. Use when creating a new skill, reviewing skill structure, optimizing a skill description, or setting up evals for skill quality.

FileKindSize
SKILL.mdskill md10.6 KB
references/evaluating-skills.mdmarkdown4.4 KB
references/optimizing-descriptions.mdmarkdown3.4 KB

references/evaluating-skills.md

View on GitHub
# Evaluating Skill Output Quality

Full eval workflow for testing whether a skill produces good outputs.

## Test Cases

Store in `evals/evals.json`:

```json
{
  "skill_name": "csv-analyzer",
  "evals": [
    {
      "id": 1,
      "prompt": "I have a CSV of monthly sales data in data/sales_2025.csv. Find the top 3 months by revenue and make a bar chart.",
      "expected_output": "A bar chart showing top 3 months by revenue with labeled axes.",
      "files": ["evals/files/sales_2025.csv"]
    }
  ]
}
```

**Writing good test prompts:**
- Start with 2-3 cases, expand after first results
- Vary phrasing (formal, casual, typos, abbreviations)
- Vary explicitness (name the domain vs. describe the need)
- Cover edge cases (malformed input, unusual requests)
- Use realistic context (file paths, column names, personal context)

## Workspace Structure

```
skill-workspace/
└── iteration-1/
    ├── eval-top-months-chart/
    │   ├── with_skill/
    │   │   ├── outputs/
    │   │   ├── timing.json
    │   │   └── grading.json
    │   └── without_skill/
    │       ├── outputs/
    │       ├── timing.json
    │       └── grading.json
    └── benchmark.json
```

## Running Evals

Each run needs a clean context (no leftover state). Use subagents or separate sessions.

For each test case, run **with skill** and **without skill** (or with previous version as baseline).

When improving an existing skill, snapshot the old version first:
```bash
cp -r <skill-path> <workspace>/skill-snapshot/
```

### Timing Data

Record after each run:
```json
{
  "total_tokens": 84852,
  "duration_ms": 23332
}
```

## Assertions

Add after seeing first outputs. Verifiable statements about what the output should contain.

Good assertions:
- "The output file is valid JSON"
- "The bar chart has labeled axes"
- "The report includes at least 3 recommendations"

Weak assertions:
- "The output is good" (too vague)
- "Uses exactly the phrase 'Total Revenue: $X'" (too brittle)

Add to `evals.json`:
```json
{
  "assertions": [
    "The output includes a bar chart image file",
    "The chart shows exactly 3 months",
    "Both axes are labeled"
  ]
}
```

## Grading

Evaluate each assertion: **PASS** or **FAIL** with concrete evidence.

```json
{
  "assertion_results": [
    {
      "text": "Both axes are labeled",
      "passed": false,
      "evidence": "Y-axis labeled 'Revenue ($)' but X-axis has no label"
    }
  ],
  "summary": {
    "passed": 3, "failed": 1, "total": 4, "pass_rate": 0.75
  }
}
```

**Grading principles:**
- Require concrete evidence for a PASS
- Review the assertions themselves; fix ones that are too easy, too hard, or unverifiable
- For comparing versions: try blind comparison (present both outputs without revealing which is which)

## Aggregating Results

Save to `benchmark.json`:
```json
{
  "run_summary": {
    "with_skill": {
      "pass_rate": { "mean": 0.83 },
      "tokens": { "mean": 3800 }
    },
    "without_skill": {
      "pass_rate": { "mean": 0.33 },
      "tokens": { "mean": 2100 }
    },
    "delta": { "pass_rate": 0.50, "tokens": 1700 }
  }
}
```

## Analyzing Patterns

- **Remove assertions that always pass in both configs** they inflate pass rate without reflecting skill value
- **Investigate assertions that always fail in both** assertion may be broken
- **Study assertions that pass with skill but fail without** understand *why*
- **Tighten instructions when results are inconsistent** add examples or more specific guidance
- **Check time/token outliers** read execution transcripts to find bottlenecks

## Human Review

Record specific, actionable feedback per test case in `feedback.json`:
```json
{
  "eval-top-months-chart": "Chart months in alphabetical order instead of chronological.",
  "eval-clean-emails": ""
}
```

Empty = passed review.

## Iteration Loop

1. Give eval signals (failed assertions + human feedback + execution transcripts) and current SKILL.md to an LLM
2. Review and apply proposed changes
3. Rerun all test cases in `iteration-<N+1>/`
4. Grade and aggregate
5. Human review. Repeat.

**Guidelines for revisions:**
- **Generalize from feedback** don't patch for specific test cases
- **Keep the skill lean** fewer, better instructions
- **Explain the *why*** reasoning-based instructions > rigid directives
- **Bundle repeated work** if agents reinvent the same logic, put it in scripts/

Stop when feedback is consistently empty or improvements plateau.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall