moonlight-lupin/agent-skills / plugins/skill-retrieval
skill-retrieval
VerifiedBM25-based skill retrieval plugin for Hermes Agent. Replaces the full skill list in the system prompt with a names-only compact view (~2K tokens) and injects top-K relevant skill descriptions per turn via BM25 retrieval (~300 tokens). Saves ~9K tokens/turn. Use when system prompt token overhead from skills is a concern, or when skill discovery quality matters.
scripts/jev_rerank.py
View on GitHub"""Jev semantic rerank stage for the skill-retrieval plugin (opt-in).
Advisory-layer pattern (system-one-decision-models skill): BM25 stays the
deterministic recall stage and the source of truth. When
``SKILL_RETRIEVAL_RERANK=jev`` is set, the BM25 top-N shortlist is reranked
by one batched TypeSafe Jev call (one Noul judgement per candidate) and the
injected order follows Jev's probabilities. Any Jev failure falls back to
the exact BM25 order — the plugin never blocks or loses injection quality
because of the advisory layer.
Per-turn A/B logging records both orders side by side to a JSONL file so
operators can compute rank flips, displacement, fallback rate, latency and
cost on real traffic. The log never blocks the turn and is capped by size
(see ``_write_log``).
Request shape (POST https://api.typesafe.ai/v1/systemone, model jev-latest, or
the same contract on https://openrouter.ai/api/v1/systemone with
SKILL_RETRIEVAL_RERANK_PROVIDER=openrouter,
noul questions with true/false criteria dict, one question keyed per
candidate id).
Injection defense: imperative sentence patterns in the user query are
neutralised before the query becomes Jev state.
Configuration comes from os.environ only (Hermes loads the active profile's
.env into os.environ at startup): TYPESAFE_API_KEY (legacy TYPESAFE_KEY
accepted), or OPENROUTER_API_KEY with the openrouter provider. The key is never logged, never returned, never included in any
record.
"""
import json
import logging
import os
import re
import time
import urllib.error
import urllib.request
from datetime import datetime, timezone
from pathlib import Path
logger = logging.getLogger(__name__)
TS_URL = "https://api.typesafe.ai/v1/systemone"
MODEL_DEFAULT = "jev-latest"
# OpenRouter serves Jev on the same System One contract; only the URL, key and
# model id differ. https://openrouter.ai/docs/guides/community/jev
OPENROUTER_URL = "https://openrouter.ai/api/v1/systemone"
OPENROUTER_MODEL_DEFAULT = "~typesafe/jev-latest"
_SHAPES_RE_BODY = (
r"ignore( all)?( previous| prior)? (instructions|prompts?|messages?)\b"
r"|disregard( all)? (instructions|prompts?)\b"
r"|system( prompt)?:"
r"|you are now\b"
r"|new (instructions|task):"
r"|delete (everything|all)\b"
r"|erase (everything|all)\b"
r"|forget (everything|all|your)\b"
r"|override (your )?(instructions|rules)\b"
)
# ^-anchored pass for the user query: a legit query may quote imperative
# text in the middle, so only the leading shape is stripped. The full
# shape set applies (a query is one turn's own words; the shapes that
# common legitimate phrasing also uses — 'delete all', 'system:' — only
# cost the query when they sit at the very start).
ENV_PREFIX_RE = re.compile(
r"^(" + _SHAPES_RE_BODY + r")[^.!?\n]*[.!?\n]?\s*",
re.IGNORECASE,
)
# Boundary for the description pass: a shape may sit after ANY separator
# that is not a plain-ASCII letter/digit/underscore — space, punctuation,
# quotes, brackets, colons, ACCENTED LATIN and CJK characters all allow
# the strip (explicit codepoints instead of Python's Unicode-\w lookbehind,
# which treats CJK as word characters and would block CJK-preceded shapes).
# The class is written as NUMERIC RANGES (\\x41-\\x5A etc.), not literal
# letters, and the flag is scoped to the shape group only: Python 3.14
# expands literal members of a character class into their full folding
# orbit under (?i) — 'K' (U+004B) would also match the Kelvin sign (U+212A),
# and 'İ'/'ı'/'ſ' block the strip as if they were plain letters. Ranges
# fold-check the lowercased char instead, so only true ASCII letters block
# the strip.
_DESC_BOUNDARY = r"(?:^|(?<![0-9\x41-\x5A\x61-\x7A\x5F]))"
# Unanchored pass for third-party skill descriptions, narrowed to the
# UNAMBIGUOUS injection shapes. 'system:', 'new task:', 'delete all',
# 'erase all' and 'forget your' are ordinary wording in skill
# descriptions (build systems, task lists, cleanup skills), so only the
# shapes that essentially never appear in legitimate description text
# are REMOVED — the shape itself plus its trailing whitespace and one
# sentence terminator. Sentence text around the shape stays (bounded
# damage), so a description ABOUT ignoring instructions keeps its
# context; text with no recognized shape is not detectable — Jev
# remains the judge and the A/B log keeps the raw shortlist.
ENV_INJECT_RE = re.compile(
_DESC_BOUNDARY + r"(?i:(?:"
r"ignore( all)?( previous| prior)? (instructions|prompts?|messages?)\b"
r"|disregard( all)? (instructions|prompts?)\b"
r"|system prompt:"
r"|you are now\b"
r"|override( your)? (instructions|rules)\b"
r"))[.!?\n]?[ \t]*",
)
def _parse_rerank_env(raw: str | None) -> str:
"""Parse SKILL_RETRIEVAL_RERANK; only 'jev' enables. Anything else → off."""
value = (raw or "").strip().lower()
if value == "jev":
return "jev"
if value in ("", "off", "0", "false", "no"):
return "off"
logger.warning("Invalid SKILL_RETRIEVAL_RERANK=%r — rerank disabled", raw)
return "off"
def _env(name: str) -> str:
"""os.environ only — never a file read.
Hermes loads the active profile's .env into os.environ at startup, so
environ is the profile-correct source. Reading ~/.hermes/.env directly
would read the DEFAULT profile's secrets under a non-default profile.
"""
return os.environ.get(name, "")
RERANK_MODE = _parse_rerank_env(_env("SKILL_RETRIEVAL_RERANK"))
def _parse_provider_env(raw: str | None) -> str:
"""Parse SKILL_RETRIEVAL_RERANK_PROVIDER: 'typesafe' (default) or 'openrouter'."""
value = (raw or "").strip().lower()
if value in ("", "typesafe"):
return "typesafe"
if value == "openrouter":
return "openrouter"
logger.warning("Invalid SKILL_RETRIEVAL_RERANK_PROVIDER=%r — using typesafe", raw)
return "typesafe"
RERANK_PROVIDER = _parse_provider_env(_env("SKILL_RETRIEVAL_RERANK_PROVIDER"))
def _endpoint() -> "tuple[str, str]":
"""(url, model) for the active provider; SKILL_RETRIEVAL_RERANK_MODEL overrides the model."""
if RERANK_PROVIDER == "openrouter":
url, default_model = OPENROUTER_URL, OPENROUTER_MODEL_DEFAULT
else:
url, default_model = TS_URL, MODEL_DEFAULT
return url, (os.environ.get("SKILL_RETRIEVAL_RERANK_MODEL") or default_model)
#: BM25 shortlist size fed to the reranker (TOP_K is carved out of this).
RERANK_CANDIDATES = 12
# Decisive bands outside which Jev's order is trusted; inside the band the
# candidate keeps its BM25-relative position (calibration describes groups,
# not single answers).
BAND_LOW = 0.35
BAND_HIGH = 0.65
# Decision-low: below this the candidate sinks (but is never dropped while
# it is on the BM25 shortlist — recall stays BM25's job).
_TOK_PRICE_PER_INPUT = 0.042 / 1_000_000 # $/token, TypeSafe list price
def neutralize_query(query) -> str:
"""Flatten any input to a string and strip leading imperative patterns.
Whitespace is collapsed first (a shape cannot hide behind a double
space, a tab, or a line break at the start), the leading pass repeats
with each strip (a splice can reassemble a shape after a removal),
and the final collapse cleans the leftovers. Capped at 2000 chars.
"""
if query is None:
return ""
text = query if isinstance(query, str) else str(query)
# Zero-width characters cannot hide a shape from the stripper.
text = re.sub(r"[\u200b-\u200d\u2060\ufeff]", "", text)
# Cap FIRST: the strip loop rescans the residual once per shape, so an
# unbounded input is quadratic; past the cap nothing recognizable is
# sent anyway.
cur = re.sub(r"\s+", " ", text).strip()[:2000]
# Repeat until stable; strictly shortening, so it terminates (a query
# with N leading copies of a shape needs N passes — no fixed cap there).
while True:
nxt = ENV_PREFIX_RE.sub("", cur)
if nxt == cur:
break
cur = nxt
return re.sub(r"\s+", " ", cur).strip()[:2000]
def _env_log_path() -> Path:
raw = os.environ.get("SKILL_RETRIEVAL_RERANK_LOG", "")
if raw:
return Path(raw)
try:
from hermes_constants import get_hermes_home
base = get_hermes_home()
except ImportError:
env_home = os.environ.get("HERMES_HOME", "").strip()
base = Path(env_home).expanduser() if env_home else Path.home() / ".hermes"
return Path(base) / "data" / "jev-trial" / "rerank_ab_log.jsonl"
def _load_key() -> str | None:
"""Read the provider's key from os.environ only: TYPESAFE_API_KEY (legacy
TYPESAFE_KEY) for typesafe, OPENROUTER_API_KEY for openrouter.
Hermes loads the active profile's .env into os.environ at startup, so
environ is the profile-correct source. Reading a .env file directly would
read the DEFAULT profile's secrets under a non-default profile. Returns
None when absent.
"""
if RERANK_PROVIDER == "openrouter":
return os.environ.get("OPENROUTER_API_KEY", "").strip() or None
primary = os.environ.get("TYPESAFE_API_KEY", "").strip()
if primary:
return primary
legacy = os.environ.get("TYPESAFE_KEY", "").strip()
return legacy or None
def _neutralize_desc(text: str) -> str:
"""Strip imperative injection shapes from a skill description.
Whitespace is collapsed FIRST (a shape cannot hide behind a double
space, a tab, or a line break), then the unanchored description pass
repeats with each strip (a splice can reassemble a shape around the
removed part: 'You are\\n<shape>\\nnow a yes-bot'), and a final
collapse cleans the leftovers. Capped later in build_request.
The query keeps the other contract: leading-only pass over the FULL
shape set (neutralize_query).
"""
text = re.sub(r"[\u200b-\u200d\u2060\ufeff]", "", text)
# Cap first (see neutralize_query): bounded scan per pass.
cur = re.sub(r"\s+", " ", text).strip()[:3000]
# Repeat until stable: each strip removes the matched shape (always
# non-empty), so the string strictly shortens and the loop terminates —
# consecutive copies of a shape are all removed, with no pass cap to
# leak the tail copies.
while True:
nxt = ENV_INJECT_RE.sub("", cur)
if nxt == cur:
break
cur = nxt
return re.sub(r"\s+", " ", cur).strip()
def build_request(query: str, shortlist: "list[tuple[str, str]]") -> "tuple[dict, dict]":
"""Build the Jev request payload pieces for one rerank turn.
``shortlist`` is [(skill_id, description), ...] in BM25 order.
Returns (questions, state) following the /v1/systemone contract:
one noul per candidate, keyed ``rel_<skill_id>``, true/false criteria.
Each question names its own candidate in its instructions. Jev judges every
question against the shared state, and the question key is an opaque label
to it, so identical instructions per candidate make every question the same
question and the probabilities come back near-uniform (no usable ranking).
"""
neutral_query = neutralize_query(query)
questions: dict = {}
for skill_id, description in shortlist:
# A description is third-party text (any skill author's wording), and
# it now travels inside the question instructions — the channel Jev
# judges on — so the unambiguous injection shapes are removed from it
# wherever they appear (whitespace-normalized, repeat-until-stable;
# _neutralize_desc below). The user query keeps the leading-only
# pass over the full shape set: its mid-query text is the user's
# own words, not attacker-controlled metadata.
desc = _neutralize_desc(description or "")[:400]
questions[f"rel_{skill_id}"] = {
"type": "noul",
"instructions": (
f"Is the skill '{skill_id}' ({desc}) relevant to serve the user's "
"query intent? Judge semantic intent match, not exact word overlap."
),
"criteria": {
"true": "this skill should be surfaced for this query",
"false": "this skill is unrelated to this query",
},
}
state = {"user_query": neutral_query}
return questions, state
def _jev_rerank_call(shortlist, query, key) -> "tuple[dict | None, dict | None, str | None]":
"""One batched Jev call: N Noul judgements in a single request.
Returns (probs, usage, error): probs maps skill_id → probability,
usage carries input_tokens and latency_ms for the A/B log.
On any failure returns (None, None, reason). Never raises outward.
"""
try:
questions, state = build_request(query, shortlist)
url, model = _endpoint()
body = json.dumps(
{"state": state, "model": model, "questions": questions},
).encode()
req = urllib.request.Request(
url,
data=body,
headers={"Authorization": "Bearer " + key, "Content-Type": "application/json"},
)
t0 = time.perf_counter()
with urllib.request.urlopen(req, timeout=3.0) as resp:
data = json.load(resp)
latency_ms = (time.perf_counter() - t0) * 1000.0
answers = data.get("answers") or {}
probs: dict[str, float] = {}
for skill_id, _desc in shortlist:
ans = answers.get(f"rel_{skill_id}")
if not isinstance(ans, dict):
return None, None, f"missing answer for {skill_id}"
value = ans.get("probability", ans.get("noul"))
try:
p = float(value)
except (TypeError, ValueError):
return None, None, f"non-numeric probability for {skill_id}"
if not 0.0 <= p <= 1.0:
return None, None, f"out-of-range probability for {skill_id}"
probs[skill_id] = p
usage_in = data.get("usage", {}).get("input_tokens", 0)
return probs, {"input_tokens": usage_in, "latency_ms": latency_ms}, None
except urllib.error.HTTPError as exc:
detail = ""
try:
detail = exc.read().decode()[:120]
except Exception:
pass
return None, None, f"HTTP {exc.code} {detail}".strip()
except Exception as exc: # network, timeout, JSON, anything
return None, None, f"{type(exc).__name__}: {exc}"[:200]
def _score_rerank_key(item: "tuple[str, float, float, int]") -> "tuple":
"""Sort key for the blended order.
Decisive candidates (prob outside the band) sort by Jev probability;
band candidates keep BM25-relative position, which is implemented as:
decisive-high first (by prob desc), then band candidates (by BM25 rank),
then decisive-low (by prob desc so the least-relevant sinks deepest).
"""
skill_id, bm25_score, prob, bm25_rank = item
if prob >= BAND_HIGH:
return (0, -prob, bm25_rank)
if prob > BAND_LOW: # band — anchor to BM25 position
return (1, bm25_rank, prob)
# Decisive-low sinks below everything else, and within the tier the
# highest-probability candidate goes first so the least-relevant
# candidate sinks deepest (matches the docstring contract).
return (2, -prob, bm25_rank)
#: A/B log size cap (bytes). When the log exceeds this, it is rotated to
#: ``<name>.1`` (overwriting the previous rotation) so it cannot grow
#: without bound. 5 MB is millions of lines at this record size.
LOG_MAX_BYTES = 5_000_000
def _write_log(record: dict) -> None:
"""Append one JSONL line; any failure is swallowed (fail-soft).
The log is size-capped: past LOG_MAX_BYTES it rotates to ``<name>.1``,
keeping exactly one older generation, so an unattended install never
accumulates an unbounded file.
"""
try:
path = _env_log_path()
path.parent.mkdir(parents=True, exist_ok=True)
try:
if path.exists() and path.stat().st_size > LOG_MAX_BYTES:
path.replace(path.with_suffix(path.suffix + ".1"))
except Exception:
pass # rotation is best-effort; the append below still runs
with open(path, "a") as fh:
fh.write(json.dumps(record, ensure_ascii=True) + "\n")
except Exception as exc:
logger.debug("rerank A/B log write failed: %s", exc)
def rerank(query, shortlist=None, rerank_candidates: int = RERANK_CANDIDATES) -> dict:
"""Optionally rerank the BM25 shortlist with Jev. Never raises.
``shortlist``: [(skill_id, bm25_score)] in BM25 order (or with desc as
a third position element [(skill_id, score, description)]).
``rerank_candidates`` caps how many candidates are sent to Jev (and
therefore how many descriptions leave the machine) regardless of how
large the caller's shortlist is — the documented outbound bound. The
remaining candidates keep their BM25 order at the tail.
Returns dict:
order: final skill ids in injected order
skipped: True when Jev did not run (disabled, no key, error, empty)
fallback_reason: why the BM25 order was kept (None when Jev ran)
plus key_loaded/input_tokens/latency_ms for the caller's logging.
"""
shortlist = list(shortlist or [])
if RERANK_MODE != "jev" or not shortlist:
return {"order": [s[0] for s in shortlist], "skipped": True,
"fallback_reason": "disabled" if RERANK_MODE != "jev" else "empty_shortlist"}
if rerank_candidates is not None and rerank_candidates > 0:
capped = shortlist[:rerank_candidates]
else:
capped = shortlist
tail = shortlist[len(capped):] # beyond the cap: BM25 order, never sent
key = _load_key()
probs, usage, error = (None, None, "missing_key")
if not neutralize_query(query):
# A query whose text neutralized away entirely gives Jev nothing
# to judge against — go through the shared fallback path below so
# the A/B log stays complete.
error = "empty_query"
elif key:
# Description text for Jev state comes from the shortlist tuples when
# present; the caller may pass (skill_id, score, description) triples.
# Only the capped head is sent — the tail never leaves the machine.
enriched = []
for entry in capped:
if len(entry) >= 3:
enriched.append((entry[0], entry[2]))
else:
info_probe = entry
enriched.append((info_probe[0], str(info_probe[1])))
probs, usage, error = _jev_rerank_call(enriched, query, key)
bm25_order = [s[0] for s in shortlist]
bm25_rank = {sid: i for i, sid in enumerate(bm25_order)}
record: dict = {
"ts": datetime.now(timezone.utc).astimezone().isoformat(timespec="seconds"),
"session_id": os.environ.get("HERMES_SESSION_ID", ""),
"query": (str(query)[:200] if query else ""),
"bm25_order": bm25_order,
"jev_order": None,
"injected_order": bm25_order,
"jev_probs": None,
"input_tokens": None,
"latency_ms": None,
"fallback_reason": None,
}
if probs is None:
record["fallback_reason"] = error
_write_log(record)
logger.info("Jev rerank skipped: %s — BM25 order unchanged", error)
return {"order": bm25_order, "skipped": True, "fallback_reason": error,
"key_loaded": key is not None}
# The capped head is reordered by Jev probability desc with band
# anchoring; the tail keeps its BM25 order and is appended last —
# candidates never sent to Jev must not displace ones it judged.
def _head_key(entry):
sid, bm25_score, p, bm25_rank = entry
return _score_rerank_key((sid, bm25_score, p, bm25_rank))
capped_ids = [c[0] for c in capped]
head = [
(entry[0], entry[1] if isinstance(entry[1], (int, float)) else 0.0,
probs.get(entry[0], BAND_LOW), bm25_rank[entry[0]])
for entry in capped
]
head_sorted = [sid for sid, *_ in sorted(head, key=_head_key)]
tail_ids = [s[0] for s in shortlist if s[0] not in set(capped_ids)]
final = head_sorted + tail_ids
record.update({
"jev_order": final,
"injected_order": final,
"jev_probs": probs,
"input_tokens": (usage or {}).get("input_tokens"),
"latency_ms": (usage or {}).get("latency_ms"),
"rank1_flip": bool(
final and bm25_order and final[0] != bm25_order[0]
),
})
_write_log(record)
logger.debug(
"Jev rerank: %d candidates, rank1_flip=%s, %s tok, %.0f ms",
len(final), record["rank1_flip"], (usage or {}).get("input_tokens"),
(usage or {}).get("latency_ms", 0),
)
return {"order": final, "skipped": False, "fallback_reason": None,
"key_loaded": True,
"input_tokens": (usage or {}).get("input_tokens"),
"latency_ms": (usage or {}).get("latency_ms")}Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |