moonlight-lupin/agent-skills / plugins/skill-retrieval
skill-retrieval
VerifiedBM25-based skill retrieval plugin for Hermes Agent. Replaces the full skill list in the system prompt with a names-only compact view (~2K tokens) and injects top-K relevant skill descriptions per turn via BM25 retrieval (~300 tokens). Saves ~9K tokens/turn. Use when system prompt token overhead from skills is a concern, or when skill discovery quality matters.
README.md
View on GitHub# skill-retrieval
BM25-based skill retrieval plugin for [Hermes Agent](https://hermes-agent.nousresearch.com). Replaces the full skill list in the system prompt with a names-only compact view (~2K tokens) and injects top-K relevant skill descriptions per turn (~300 tokens), saving ~9K tokens/turn while keeping skills discoverable by name. Those figures were measured on a Hermes install with ~300 skills; the saving scales with your own skill count.
This is a **Hermes Agent** plugin. It is not a Claude Code plugin and will not load in Claude Code — that runtime has no `pre_llm_call` event, no Python `register()` entry point, and reads `.claude-plugin/plugin.json` rather than `plugin.yaml`. Requires Hermes Agent >=0.21.1.
See [SKILL.md](SKILL.md) for full architecture, token measurements, how it works, performance, limitations, and how to verify a healthy install.
## Contents
- Installation
- Configuration
- Read-only core references
- Uninstall
- Data handling (Jev rerank)
- License
## Installation
```bash
# From the agent_skills repo root
ln -s "$(pwd)/plugins/skill-retrieval" ~/.hermes/plugins/skill-retrieval
# Named profile: the plugin reads its own directory relative to the plugin
# folder, and keys/paths resolve under $HERMES_HOME (set per profile) — no
# cross-profile fallback. Copy the plugin into the profile's plugins/ dir
# and set any needed keys in that profile's environment (Hermes loads the
# profile's .env into os.environ at startup).
# Dependencies (Hermes Python env)
# pyyaml is the only dependency — usually already present in a Hermes env.
# On CPython 3.14 the plugin uses the host's hermes_yaml facade instead.
pip install pyyaml
```
User plugins are opt-in — Hermes discovers the directory but does not load it until you enable it:
```bash
hermes plugins enable skill-retrieval
```
Then restart the Hermes session so the plugin's `register()` runs.
## Configuration
| Setting | Default | Override |
|---------|---------|----------|
| Top-K results | `6` | `SKILL_RETRIEVAL_TOP_K` env var |
| Injected results | `TOP_K` | `SKILL_RETRIEVAL_INJECT_K` env var; a pool wider than the injected set (e.g. `TOP_K=12`, `INJECT_K=6`) lets the Jev rerank filter, not just reorder |
| System prompt compaction | enabled | `SKILL_RETRIEVAL_COMPACT=0` disables compaction but keeps retrieval injection |
| BM25 k1 | `1.5` | edit `scripts/bm25_retriever.py` |
| BM25 b | `0.75` | edit `scripts/bm25_retriever.py` |
| Jev rerank | off | `SKILL_RETRIEVAL_RERANK=jev` enables; key `TYPESAFE_API_KEY` (legacy `TYPESAFE_KEY` accepted) from `os.environ` |
| Session rerank cache | off | `SKILL_RETRIEVAL_RERANK_SESSION_CACHE=1` reuses a named session's confirmed candidate workset and avoids redundant Jev calls while BM25 stays inside it |
| Rerank provider | `typesafe` | `SKILL_RETRIEVAL_RERANK_PROVIDER=openrouter` calls Jev through [OpenRouter](https://openrouter.ai/docs/guides/community/jev) with `OPENROUTER_API_KEY` |
| Rerank model | `jev-latest` (typesafe), `~typesafe/jev-latest` (openrouter) | `SKILL_RETRIEVAL_RERANK_MODEL`, e.g. `typesafe/jev-1.14` to pin a version |
| Rerank A/B log | `$HERMES_HOME/data/jev-trial/rerank_ab_log.jsonl` (rotates to `.1` past 5 MB) | `SKILL_RETRIEVAL_RERANK_LOG` env var |
```bash
export SKILL_RETRIEVAL_TOP_K=8
export SKILL_RETRIEVAL_INJECT_K=6
export SKILL_RETRIEVAL_COMPACT=0 # retrieval-only mode; the skills prompt is left unchanged (the builder is still wrapped to record tool capabilities)
export SKILL_RETRIEVAL_RERANK=jev # optional: rerank the top-K with the Jev system-one model
export SKILL_RETRIEVAL_RERANK_SESSION_CACHE=1 # optional: reuse a named session's confirmed workset
export SKILL_RETRIEVAL_RERANK_PROVIDER=openrouter # optional: call Jev through OpenRouter with OPENROUTER_API_KEY instead of a TypeSafe key
```
### Jev rerank
Optional, off by default. When enabled, the BM25 top-N shortlist is reranked by the Jev system-one judgment model (`api.typesafe.ai/v1/systemone`, model `jev-latest`). It is fail-soft: a 3.0 s timeout with no retries, and any error falls back to the BM25 order. The key is read from `os.environ` only (Hermes loads the active profile's `.env` into environ at startup — no cross-profile file reads). Every query is logged to the A/B log path; `scripts/rerank_ab_report.py` summarizes the log locally. See **Data handling** below for exactly what leaves the machine.
**Evaluate over a period of time before deciding to keep it.** A single session cannot show whether the rerank helps, so the practical benefit is hard to judge turn by turn. Enable the rerank, use it for real work for at least 1-2 weeks, then review the A/B log with `scripts/rerank_ab_report.py` — look at how often the Jev order differs from the BM25 order (`rank1_flip`), what the per-turn probability spread looks like, and whether the flips are right. If the spread is consistently tiny or the flips look arbitrary, turn it off.
### Session rerank cache
Optional and off by default. This mode is intended for conversations whose
topic stays coherent for several turns. The first successful Jev rerank freezes
that named session's BM25 candidate pool as a workset. On later turns, local
BM25 still runs and determines the current order. If every candidate that would
be injected is already in the workset (or was selected by a later successful
rerank), the external Jev call is skipped. A new would-be-injected candidate
triggers Jev again; only a successful Jev result can confirm it for reuse.
The cache never changes the system prompt, never reuses anonymous requests,
and never crosses session ids. It is bounded to 256 sessions and is cleared on
skill installation, removal, or other skill-catalog mutations. On a cache hit,
the current BM25 order is injected, so enable this only when avoiding repeated
network reranks is worth that semantic tradeoff.
## Read-only core references
The plugin imports ten Hermes core helpers **read-only** (never assigned, never wrapped). From `agent.prompt_builder`:
- `_current_session_platform_hint` — session platform for cache keying and visibility gating
- `_parse_skill_file`, `_build_snapshot_entry` — SKILL.md frontmatter/description parsing for the corpus
- `_skill_should_show` — per-skill visibility gating (tools, toolsets, platform)
- `extract_skill_conditions` — condition extraction feeding `_skill_should_show`
From `agent.skill_utils`:
- `get_all_skills_dirs`, `get_project_skills_dirs` — skill root resolution (profile, external dirs)
- `get_disabled_skill_names` — disabled-set gating
- `iter_project_skill_files`, `iter_skill_index_files` — SKILL.md file enumeration
- (`skill_matches_platform` is consulted via `getattr` when present)
These are read-only imports disclosed per the catalog's rule-9 note. If a Hermes upgrade changes their signatures, the corpus loader is guarded: any mid-scan exception falls back to the standalone loader, so the index is never silently empty while compaction strips descriptions. If a future change needs an assignment or wrap there, that counts against rule 9's strict reading and warrants a core seam request instead.
In `codex_app_server` mode the prompt goes to Codex as `developerInstructions` and bypasses `llm_request`, so that mode gets no compaction and no snapshot, while turn-1 fail-open still retrieves.
## Uninstall
1. Disable the plugin (`hermes plugins disable skill-retrieval`), then delete the plugin folder from `~/.hermes/plugins/` with your file manager or shell.
2. Restart the Hermes session
3. The system prompt reverts on restart
The middleware and hook are session-scoped, so removal is otherwise instant.
## Data handling (Jev rerank)
The rerank is **off by default**. With no `SKILL_RETRIEVAL_RERANK=jev` set,
the plugin makes **no network calls** and writes nothing.
When you opt in (`SKILL_RETRIEVAL_RERANK=jev` + a key present):
- **Sent to `api.typesafe.ai`** (normally one POST per retrieval turn; with
`SKILL_RETRIEVAL_RERANK_SESSION_CACHE=1`, confirmed named-session turns can
skip the POST; model
`jev-latest`): the user message flattened, stripped of leading imperative
shapes over the full shape set (shape plus its following sentence), capped
at 2000 characters, plus one
question per top-12 shortlisted skill naming that skill's id and its
description — the description is whitespace-normalized and unambiguous
imperative shapes are removed wherever they appear (the shape itself plus
its trailing whitespace and terminator; sentence text around it stays;
ordinary wordings like 'new task:', 'delete all' are kept), capped at 400
characters. BM25 scores never leave the machine. The response
carries one relevance probability per candidate.
- **Key**: `TYPESAFE_API_KEY` (legacy `TYPESAFE_KEY` accepted), read from
`os.environ` only — Hermes loads the active profile's `.env` into
environ at startup, so no other profile's file is ever read. The key is
never logged, never written, never included in any record.
- **Through OpenRouter** (`SKILL_RETRIEVAL_RERANK_PROVIDER=openrouter`): the
same payload goes to `https://openrouter.ai/api/v1/systemone`, which serves
Jev on the same System One contract and forwards it to TypeSafe. The key is
`OPENROUTER_API_KEY`, also from `os.environ` only. Each provider reads only
its own key, so a TypeSafe key is never sent to OpenRouter and vice versa.
- **Local log**: each reranked turn appends one JSONL line to
`$HERMES_HOME/data/jev-trial/rerank_ab_log.jsonl` (override:
`SKILL_RETRIEVAL_RERANK_LOG`) containing the timestamp, session id, a
200-character query excerpt, the BM25 order, the Jev order, probabilities,
token count, and latency. The log rotates to `<name>.1` past 5 MB, so it
never grows without bound. `scripts/rerank_ab_report.py` summarizes it
locally; no log data is uploaded anywhere.
## License
MIT. See [LICENSE](LICENSE) for the upstream SR-Agents notice and this project's notice.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |