richfrem/agent-plugins-skills / plugins/agent-memory/.apm/skills/managing-obsidian-vaults
managing-obsidian-vaults
VerifiedInteractive vault, wiki and Super-RAG setup in the main conversation. Use for "set up my wiki", "initialize the full knowledge stack" or combined RLM/vector/wiki setup. Manage Obsidian vaults and LLM wikis: initialize vaults, author Markdown and wikilinks, safely edit notes and Bases, create Canvas diagrams, traverse graphs, build concept wikis, distill summary/bullets/deep layers, query and lint knowledge.
scripts/distill_wiki.py
View on GitHub#!/usr/bin/env python
"""
distill_wiki.py
=====================================
Purpose:
Cheap-model RLM distillation orchestrator for the Obsidian Wiki Engine.
Iterates over all wiki nodes needing RLM summaries, auto-detects the
cheapest available LLM CLI, and generates three summary layers per concept:
summary.md, bullets.md, deep.md.
Distillation is performed entirely within this plugin — no cross-plugin
script calls. RLM summaries are written to a configurable cache directory
(default: {wiki_root}/rlm/) or to a shared path under .agent/learning/
when --rlm-cache-dir is set to point there.
Fallback chain (strict — Ollama is fully deprecated):
1. copilot -> gpt-5-mini
2. claude -> claude-haiku-4-5
3. gemini -> gemini-3-flash-preview
Layer: Distill / Wiki
Usage:
python ./scripts/distill_wiki.py --wiki-root /path/to/wiki-root
python ./scripts/distill_wiki.py --wiki-root /path/to/wiki-root --source arch-docs
python ./scripts/distill_wiki.py --wiki-root /path/to/wiki-root --engine claude
python ./scripts/distill_wiki.py --wiki-root /path/to/wiki-root --dry-run
python ./scripts/distill_wiki.py --wiki-root /path/to/wiki-root \\
--rlm-cache-dir /path/to/project/.agent/learning/rlm_wiki_cache
Related:
- raw_manifest.py (WikiSourceConfig + agent-memory.json)
- wiki_builder.py (consumes the rlm/ summaries it produces)
- audit.py (reports missing summaries)
Key Input Dependencies:
- Wiki node files under {wiki_root}/wiki/*.md
- One of copilot/claude/gemini CLIs available on PATH
"""
import sys
import os
import shutil
import argparse
import subprocess
from pathlib import Path
from typing import List, Optional, Tuple
SCRIPT_DIR = Path(__file__).resolve().parent
if str(SCRIPT_DIR) not in sys.path:
sys.path.insert(0, str(SCRIPT_DIR))
from raw_manifest import now_iso
# ─── ENGINE REGISTRY (strict cheap-model only) ────────────────────────────────
# Ollama is fully deprecated for distillation tasks.
ENGINE_PRIORITY: List[Tuple[str, str]] = [
("copilot", "gpt-5-mini"),
("claude", "claude-haiku-4-5"),
("gemini", "gemini-3-flash-preview"),
]
# ─── PATH AUGMENTATION (stripped PATH in some environments) ──────────────────
_EXTRA_PATHS = [
"/opt/homebrew/bin",
"/usr/local/bin",
os.path.expanduser("~/.local/bin"),
os.path.expanduser("~/.npm-global/bin"),
os.path.expanduser("~/n/bin"),
]
for _p in _EXTRA_PATHS:
if _p not in os.environ.get("PATH", "") and Path(_p).exists():
os.environ["PATH"] = _p + os.pathsep + os.environ.get("PATH", "")
def detect_engine(override: Optional[str] = None) -> Tuple[str, str]:
"""
Detect the cheapest available LLM CLI on PATH.
Args:
override: Force a specific engine name (e.g. 'claude').
Returns:
(engine_name, model_name) tuple.
Raises:
SystemExit: If override is unknown or no engine is found.
"""
if override:
for engine, model in ENGINE_PRIORITY:
if engine == override.lower():
if shutil.which(engine):
return (engine, model)
print(f"[ERROR] Engine '{override}' not found on PATH.")
print(f" Install: https://github.com/cli/{override}")
sys.exit(1)
print(f"[ERROR] Unknown engine '{override}'. Choose from: {[e for e, _ in ENGINE_PRIORITY]}")
sys.exit(1)
for engine, model in ENGINE_PRIORITY:
if shutil.which(engine):
print(f"[ENGINE] Using: {engine} ({model})")
return (engine, model)
print("[ERROR] No LLM CLI found on PATH.")
print(" Install one of:")
for engine, model in ENGINE_PRIORITY:
print(f" {engine} (model: {model})")
sys.exit(1)
def build_prompt(layer: str, concept: str, content: str) -> str:
"""
Build the distillation prompt for the given layer.
Args:
layer: One of 'summary', 'bullets', 'deep'.
concept: The concept slug being distilled.
content: The raw wiki node content excerpt.
Returns:
Prompt string ready to send to the LLM.
"""
base = f"Concept: {concept}\n\n{content}\n\n"
if layer == "summary":
return base + (
"Write a concise 1-5 sentence summary of this concept. "
"Focus on the most important idea. Plain prose, no headings, no bullets."
)
elif layer == "bullets":
return base + (
"List 6-10 key ideas from this concept as concise bullet points. "
"Use '- ' prefix. No headings, no preamble."
)
elif layer == "deep":
return base + (
"Write a thorough multi-paragraph distillation of this concept. "
"Cover the main ideas, important details, and relationships to related concepts. "
"Use markdown headings where helpful."
)
return base + "Summarize this content."
def _build_llm_cmd(engine: str, model: str, prompt: str) -> Optional[Tuple[List[str], str]]:
"""Build the (subprocess command, stdin text) pair for the given engine, or None if unknown."""
if engine == "claude":
return (["claude", "--model", model, "-p", prompt, "--no-session-persistence"], "")
if engine == "gemini":
return (["gemini", "--model", model, "-p", prompt], "")
if engine == "copilot":
return (["copilot", "--model", model], prompt)
return None
def call_llm(
engine: str,
model: str,
prompt: str,
timeout: int = 120,
) -> Optional[str]:
"""
Call the specified LLM CLI with the given prompt.
Args:
engine: CLI name ('copilot', 'claude', 'gemini').
model: Model identifier string.
prompt: Full prompt payload.
timeout: Subprocess timeout in seconds.
Returns:
Stripped LLM response string, or None on error.
"""
built = _build_llm_cmd(engine, model, prompt)
if built is None:
return None
cmd, stdin_text = built
try:
proc = subprocess.run(
cmd,
input=stdin_text or None,
capture_output=True,
text=True,
timeout=timeout,
shell=(sys.platform == "win32"),
)
except FileNotFoundError:
print(f"[ERROR] CLI '{engine}' not found on PATH.")
return None
except subprocess.TimeoutExpired:
print(f"[ERROR] {engine} timed out after {timeout}s.")
return None
if proc.returncode != 0 or not proc.stdout.strip():
print(f"[ERROR] {engine} returned exit code {proc.returncode}")
if proc.stderr:
print(proc.stderr[:300])
return None
return proc.stdout.strip()
def _resolve_rlm_cache_dir(wiki_root: Path, rlm_cache_dir: Optional[Path] = None) -> Path:
"""
Resolve the RLM cache directory.
Priority:
1. Explicit --rlm-cache-dir argument (e.g. .agent/learning/rlm_wiki_cache)
2. Default: {wiki_root}/rlm/
Using --rlm-cache-dir allows the wiki engine to write summaries into the
same .agent/learning/ tree as rlm-factory, avoiding duplicate storage.
"""
if rlm_cache_dir:
return Path(rlm_cache_dir).resolve()
return wiki_root / "rlm"
def write_rlm_layer(
concept: str,
layer: str,
content: str,
rlm_cache_dir: Path,
) -> None:
"""
Write one RLM summary layer file to {rlm_cache_dir}/{concept}/{layer}.md.
Args:
concept: Concept slug.
layer: Layer name ('summary', 'bullets', 'deep').
content: Distilled text to write.
rlm_cache_dir: Resolved RLM cache directory.
"""
rlm_dir = rlm_cache_dir / concept
rlm_dir.mkdir(parents=True, exist_ok=True)
layer_path = rlm_dir / f"{layer}.md"
layer_content = "\n".join([
"---",
f'concept: "{concept}"',
f'layer: "{layer}"',
f'generated_at: "{now_iso()}"',
"---",
"",
f"# {layer.title()}",
"",
content,
"",
])
layer_path.write_text(layer_content, encoding="utf-8")
def distill_concept(
concept: str,
wiki_root: Path,
engine: str,
model: str,
layers: List[str],
rlm_cache_dir: Path,
dry_run: bool = False,
) -> bool:
"""
Distill all requested layers for one concept.
Reads the wiki node file as source content, calls the LLM for each layer,
and writes the result to {rlm_cache_dir}/{concept}/
Args:
concept: Concept slug (matches wiki/{concept}.md filename).
wiki_root: Root of the wiki output directory.
engine: LLM CLI to use.
model: Model identifier.
layers: List of layer names to generate.
rlm_cache_dir: Resolved RLM cache directory.
dry_run: If True, build prompts but do not call LLM.
Returns:
True on success, False if the wiki node file is missing.
"""
wiki_node = wiki_root / "wiki" / f"{concept}.md"
if not wiki_node.exists():
print(f" [SKIP] Wiki node not found: {wiki_node.name}")
return False
content = wiki_node.read_text(encoding="utf-8")
for layer in layers:
prompt = build_prompt(layer, concept, content[:3000])
if dry_run:
print(f" [DRY] Would distill {concept}/{layer}.md via {engine}")
continue
print(f" [RUN] {concept}/{layer}.md via {engine}...", end=" ", flush=True)
result = call_llm(engine, model, prompt)
if result:
write_rlm_layer(concept, layer, result, rlm_cache_dir)
print(f"OK ({len(result)} chars)")
else:
print("FAILED")
return True
def get_concepts_needing_distillation(
wiki_root: Path,
layers: List[str],
rlm_cache_dir: Path,
source_name: Optional[str] = None,
) -> List[str]:
"""
Find all wiki node concepts that are missing one or more RLM layers.
Args:
wiki_root: Root of the wiki output directory.
layers: Layer names to check for.
rlm_cache_dir: Resolved RLM cache directory.
source_name: Optional source filter (not yet implemented at this level).
Returns:
Sorted list of concept slugs needing distillation.
"""
wiki_dir = wiki_root / "wiki"
if not wiki_dir.exists():
return []
needs_distill = []
for node_file in sorted(wiki_dir.glob("*.md")):
if node_file.name.startswith("_"):
continue
concept = node_file.stem
concept_rlm_dir = rlm_cache_dir / concept
missing = any(not (concept_rlm_dir / f"{layer}.md").exists() for layer in layers)
if missing:
needs_distill.append(concept)
return needs_distill
def main() -> None:
"""Parse CLI arguments and run the distillation pipeline."""
parser = argparse.ArgumentParser(
description="Distill wiki nodes into RLM summary layers using cheap cloud LLMs"
)
parser.add_argument("--wiki-root", required=True, help="Path to the wiki root directory")
parser.add_argument("--source", default=None, help="Distill one named source only (filter by source label)")
parser.add_argument("--engine", default=None, choices=["copilot", "claude", "gemini"],
help="Force a specific LLM CLI engine")
parser.add_argument("--layers", default="summary,bullets,deep",
help="Comma-separated layers to generate (default: summary,bullets,deep)")
parser.add_argument(
"--rlm-cache-dir",
default=None,
help=(
"Override the RLM cache directory (default: {wiki-root}/rlm). "
"Set to .agent/learning/rlm_wiki_cache to colocate with rlm-factory."
),
)
parser.add_argument("--dry-run", action="store_true",
help="Show what would be distilled without calling the LLM")
args = parser.parse_args()
wiki_root = Path(args.wiki_root).resolve()
layers = [layer.strip() for layer in args.layers.split(",")]
engine, model = detect_engine(args.engine)
rlm_cache_dir = _resolve_rlm_cache_dir(wiki_root, args.rlm_cache_dir)
concepts = get_concepts_needing_distillation(wiki_root, layers, rlm_cache_dir)
if not concepts:
print("[OK] All wiki nodes already have RLM summaries.")
return
print(f"\n[DISTILL] {len(concepts)} concepts need distillation")
print(f" Engine : {engine} ({model})")
print(f" Layers : {layers}")
print(f" Wiki : {wiki_root}")
print(f" RLM cache: {rlm_cache_dir}\n")
success = 0
for concept in concepts:
ok = distill_concept(
concept, wiki_root, engine, model, layers, rlm_cache_dir,
dry_run=args.dry_run,
)
if ok:
success += 1
action = "planned" if args.dry_run else "completed"
print(f"\n[DONE] Distillation {action}: {success}/{len(concepts)} concepts.")
if __name__ == "__main__":
main()
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |