Skip to content

moonlight-lupin/agent-skills / plugins/skill-retrieval

skill-retrieval

Verified

BM25-based skill retrieval plugin for Hermes Agent. Replaces the full skill list in the system prompt with a names-only compact view (~2K tokens) and injects top-K relevant skill descriptions per turn via BM25 retrieval (~300 tokens). Saves ~9K tokens/turn. Use when system prompt token overhead from skills is a concern, or when skill discovery quality matters.

---
name: skill-retrieval
description: >-
  BM25-based skill retrieval plugin for Hermes Agent. Replaces the full skill
  list in the system prompt with a names-only compact view (~2K tokens) and
  injects top-K relevant skill descriptions per turn via BM25 retrieval (~300
  tokens). Saves ~9K tokens/turn. Use when system prompt token overhead from
  skills is a concern, or when skill discovery quality matters.
license: MIT
metadata:
  version: 0.7.0
  author: moonlight-lupin
  platforms: [linux, macos, windows]
  tags: [bm25, skill-retrieval, system-prompt, token-optimization, plugin]
  hermes:
    plugin_type: hook
    hooks: [pre_llm_call]
---

# Skill Retrieval

This is a **Hermes Agent** plugin. It is not a Claude Code plugin and will
not load in Claude Code — that runtime has no `pre_llm_call` event, no Python
`register()` entry point, and reads `.claude-plugin/plugin.json` rather than
`plugin.yaml`. Requires Hermes Agent >=0.21.1 (`requires_hermes` in
`plugin.yaml`).

BM25-based progressive disclosure for Hermes Agent skills. Instead of dumping
every skill description into the system prompt (~11.5K tokens), this plugin
keeps a compact names-only index and injects only the top-K relevant
descriptions per turn.

## What it does

Two-phase progressive disclosure:

1. **Phase 1 — System prompt compaction** (an `llm_request` middleware):
   Rewrites the outgoing request's `<available_skills>` block so it lists
   skill names only (descriptions stripped). All skills remain discoverable
   by name (~2K tokens instead of ~11.5K). The system prompt Hermes builds
   is never modified — prompt caching is unaffected.

2. **Phase 2 — Per-turn BM25 retrieval** (`pre_llm_call` hook): Tokenizes the
   user message, ranks active skill descriptions with BM25 Okapi, and injects
   the top-K matches (~300 tokens) as context appended after the user
   message.

## Architecture

```
Each turn (llm_request middleware)
    │
    ▼
Phase 1: rewrite <available_skills> → names only (~2K tokens, request only)

Each turn (pre_llm_call)
    │
    ▼
Phase 2: BM25Index.retrieve(user_message, top_k)
    └── inject "## Retrieved Skills ..." into user message (~300 tokens)
```

The BM25 index is built lazily on the first turn from standalone skills
(`~/.hermes/skills`) and plugin-bundled skills (`~/.hermes/plugins/*/skills`),
then cached per Hermes home, tool capability snapshot, session platform and
resolved disabled-skill set (the same inputs Hermes keys its own skills prompt
cache on). The cache is rebuilt without a restart when Hermes clears its skills
prompt cache (`skill_manage` create/patch/delete, hub install, skill toggles)
or when a cheap on-disk manifest changes (skill root and category dir mtimes,
top-level `SKILL.md` files, `config.yaml`).
Retrieval uses a pure-stdlib inverted index (term → posting list of
precomputed BM25 weights) and is sub-millisecond for ~200 skills.

## Token savings

| Stage | Tokens (approx.) |
|-------|------------------|
| Before (full skill list in system prompt) | ~11.5K |
| After — names-only system prompt | ~2.0K |
| After — per-turn top-K descriptions | ~0.3K |
| **Net per turn** | **~2.3K** (~9K saved) |

Measured on a Hermes install with ~300 skills; savings scale with skill count.

## Installation

Copy or symlink this directory into the Hermes plugins folder:

```bash
# From this repo
ln -s "$(pwd)/plugins/skill-retrieval" ~/.hermes/plugins/skill-retrieval

# Or copy
cp -r plugins/skill-retrieval ~/.hermes/plugins/skill-retrieval
```

User plugins are opt-in: Hermes discovers the directory but does not load it
until it is enabled (this adds `skill-retrieval` to `plugins.enabled` in
`~/.hermes/config.yaml`):

```bash
hermes plugins enable skill-retrieval
```

Restart the agent session so `register()` runs — it registers the
`llm_request` middleware and the `pre_llm_call` hook.

Dependencies (install into the Hermes Python env if missing):

```bash
pip install pyyaml
```

On CPython 3.14 prefer the host's `hermes_yaml` facade (ruamel.yaml) — the
plugin resolves it automatically at runtime, falling back to PyYAML. If both
are missing it logs one warning and keeps running with an empty index.

## Configuration

| Setting | Default | How to set |
|---------|---------|------------|
| `TOP_K` | `6` | Env var `SKILL_RETRIEVAL_TOP_K` |
| `INJECT_K` | `TOP_K` | Env var `SKILL_RETRIEVAL_INJECT_K`; how many of the retrieved candidates are injected. A pool wider than the injected set (e.g. `TOP_K=12`, `INJECT_K=6`) lets the Jev rerank filter, not just reorder. Values above `TOP_K` are clamped. |
| System prompt compaction | enabled | Set `SKILL_RETRIEVAL_COMPACT=0` to disable compaction while keeping BM25 retrieval injection |
| BM25 `k1` | `1.5` | Constant in `scripts/bm25_retriever.py` |
| BM25 `b` | `0.75` | Constant in `scripts/bm25_retriever.py` |
| Jev rerank | off | Set `SKILL_RETRIEVAL_RERANK=jev` to enable; key in `TYPESAFE_API_KEY` (legacy `TYPESAFE_KEY` accepted), read from `os.environ` |
| Session rerank cache | off | Set `SKILL_RETRIEVAL_RERANK_SESSION_CACHE=1` to reuse a named session's confirmed candidate workset and skip redundant Jev calls |
| Rerank provider | `typesafe` | `SKILL_RETRIEVAL_RERANK_PROVIDER=openrouter` uses `https://openrouter.ai/api/v1/systemone` with `OPENROUTER_API_KEY` |
| Rerank log path | `$HERMES_HOME/data/jev-trial/rerank_ab_log.jsonl` (rotates past 5 MB) | Env var `SKILL_RETRIEVAL_RERANK_LOG` |

```bash
export SKILL_RETRIEVAL_TOP_K=8
export SKILL_RETRIEVAL_INJECT_K=6
export SKILL_RETRIEVAL_COMPACT=0
export SKILL_RETRIEVAL_RERANK=jev
export SKILL_RETRIEVAL_RERANK_SESSION_CACHE=1
```

### Jev rerank (optional, off by default)

When `SKILL_RETRIEVAL_RERANK=jev`, the BM25 top-N shortlist is reranked by the
Jev system-one judgment model (`POST https://api.typesafe.ai/v1/systemone`,
model `jev-latest`). Behavior:

- **Fail-soft** — 3.0 s timeout, no retries; on any error the request logs a
  warning and BM25's original order stands. Retrieval never blocks on Jev.
- **Env-only config** — all settings come from `os.environ` (never a file
  read): `TYPESAFE_API_KEY` (legacy `TYPESAFE_KEY` accepted), or
  `OPENROUTER_API_KEY` with `SKILL_RETRIEVAL_RERANK_PROVIDER=openrouter`
  (same contract, served through OpenRouter). Hermes loads
  the active profile's `.env` into environ at startup, so no cross-profile
  file is ever read.
- **A/B log** — every query records the timestamp, session id, a 200-char
  query excerpt, both orders, probabilities, tokens and latency to the log
  path above; the log rotates past 5 MB; `scripts/rerank_ab_report.py`
  summarizes it locally.
- **Outbound cap** — at most `RERANK_CANDIDATES` (default 12) descriptions
  are sent per call, even when the caller's shortlist is longer; the
  remainder keeps BM25 order at the tail and never leaves the machine.

### Session rerank cache (optional, off by default)

This mode targets phase-continuous conversations. The first successful Jev
rerank freezes the named session's BM25 candidate pool as a workset. Local
BM25 still runs on every turn. When the candidates that would be injected are
already in the workset or in the session's confirmed fallback set, the plugin
uses the current BM25 order and skips the external rerank. A candidate outside
those sets triggers Jev; it is added to the fallback set only when a successful
Jev result selects it for injection.

The state is isolated by non-empty session id, bounded to 256 sessions, and
cleared on skill-catalog mutation. Anonymous requests never reuse it. This
optimization does not rewrite or mutate Hermes' system prompt.

## Verify it's working

Phase 1 silently no-ops outside a full Hermes runtime, and the BM25 index can
silently empty. After restart, check the Hermes logs.

**Healthy start — look for these log lines:**

- `BM25 index built: N docs …`
- `Skill retrieval plugin registered (top_k=…, compact=true, rerank_session_cache=…)`

**Degraded — these warnings mean it's not working:**

- `Cannot locate prompt_builder — compaction skipped` (Phase 1 failed; Phase 2
  still runs for anonymous sessions, but named sessions skip injection because
  no capability snapshot is recorded)
- `No active skills found for BM25 index` (index is empty — zero retrieval injection)

## How it works

- **Tokenizer** — lowercases text, strips punctuation, splits on whitespace.
  Tokens with CJK-script characters are split into bigrams (for CJK-script
  runs; a 3+-character run is replaced by its bigrams): CJK Unified Ideographs
  incl. Extension A and B, Japanese kana (fullwidth and halfwidth), and hangul
  syllables — so Chinese, Japanese and Korean queries can match non-English
  corpus text, and a mixed token like `用python写` keeps the `python` term.
- **Corpus** — each skill becomes `"name: description"` plus its frontmatter
  `triggers:` list and `metadata.hermes.tags` (when present) and — when the
  SKILL.md body contains CJK-script characters — the space-joined CJK runs of
  the body's first ~3000 characters, capped at 200 characters total. With the
  body in
  the corpus a CJK query can match skill *content*, not just the (usually
  English) description. Trigger phrases, tags and the CJK runs participate in
  retrieval scoring but only the
  description is injected into the prompt. Disabled skills from
  `~/.hermes/config.yaml` are skipped.
- **Index** — BM25 Okapi TF saturation + Lucene IDF
  `log(1 + (N-df+0.5)/(df+0.5))` (always positive, so small corpora and common
  terms still score), stored as an inverted index:
  ``dict[str, list[tuple[int, float]]]`` mapping each term to a posting list of
  (doc_index, precomputed BM25 weight).
- **Retrieve** — for each unique query token present in the index, walk its
  posting list and accumulate scores; sort by descending score (score > 0 only).

## Performance

- Index built on first use and cached (~8 ms for 200 skills on a CPU-only
  VM). Each later turn only stats the skill roots, their immediate
  subdirectories and `config.yaml` to validate the cache. A zero-skill
  install caches the empty result too (one warning, no rescans).
- Retrieval is sub-millisecond (~0.03 ms mean for 200 skills). The inverted
  index touches only documents that share a query term — no full-corpus scan.
- No compiled dependencies. The plugin uses only the Python standard library
  (plus pyyaml for config/frontmatter parsing). This removes a 154 MB
  numpy/scipy install and a ~573 ms import cost, which matters for subprocess
  spawning paths (e.g. a Claude Code `UserPromptSubmit` variant).
- Failures in the hook return `None` (no injection) so the agent keeps working.

## Dependencies

- `pyyaml` — declared in `plugin.yaml` (`pip_dependencies`), so
  `hermes plugins install skill-retrieval` installs it. On CPython 3.14 the
  host's `hermes_yaml` facade is used instead when PyYAML is unavailable.

## Limitations

- BM25 is **lexical**, not semantic. Paraphrased queries that share few tokens
  with a skill's description may rank poorly even when the intent matches.
- Descriptions longer than 200 characters are truncated in the injected block;
  use `skill_view(name)` for the full skill body.
- Compaction uses Hermes's read-only `agent.prompt_builder` helpers; if they
  cannot be imported, Phase 1 is skipped and retrieval falls back to the
  standalone loader.
- A named session is only injected once its system prompt has been built in
  this process (that build records the session's tool capabilities). A session
  restored after a restart without a rebuild gets no injection rather than
  risking skills Hermes hides from it.
- A content-only edit of a nested `SKILL.md` (`category/skill/SKILL.md`)
  made outside Hermes (e.g. in an editor) changes no directory mtime, so it is
  picked up only after Hermes clears its skills prompt cache or the agent
  restarts. Edits through `skill_manage`, and any added/removed skill or
  config change, are picked up on the next turn.
- Phase 1 depends on Hermes internals (`agent.prompt_builder`) and can break
  on a Hermes upgrade.
- BM25 top-1 precision is soft: the best-matching skill is often not rank 1,
  though it usually lands within the first few results. Ranking depends entirely
  on your own corpus and how its descriptions are worded, so `TOP_K` below ~5 is
  not recommended.
- **Compaction** rewrites the request's `<available_skills>` block using
  read-only imports from `agent.prompt_builder` and `agent.skill_utils`
  (all ten disclosed in the README); if those change signature, any
  mid-scan exception falls back to the standalone loader, so the corpus
  is never silently empty.
- The stdlib index computes in float64 (the previous scipy version used
  float32). Equal-scoring skills may order differently than before. This is
  harmless — the scores are genuine ties (~1e-6 difference) — but it is a real
  behaviour delta from the scipy version.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall