Skip to content

richfrem/agent-plugins-skills / plugins/agent-memory/.apm/skills/managing-obsidian-vaults

managing-obsidian-vaults

Verified

Interactive vault, wiki and Super-RAG setup in the main conversation. Use for "set up my wiki", "initialize the full knowledge stack" or combined RLM/vector/wiki setup. Manage Obsidian vaults and LLM wikis: initialize vaults, author Markdown and wikilinks, safely edit notes and Bases, create Canvas diagrams, traverse graphs, build concept wikis, distill summary/bullets/deep layers, query and lint knowledge.

FileKindSize
SKILL.mdskill md5.9 KB
assets/templates/config.yaml.templateother556 B
assets/templates/raw-sources.json.exampleother862 B
assets/templates/wiki-node.template.mdmarkdown410 B
evals/README.mdmarkdown569 B
evals/evals.jsonconfig23.4 KB
evals/legacy/obsidian-bases-manager/evals.jsonconfig692 B
evals/legacy/obsidian-bases-manager/results.tsvother301 B
evals/legacy/obsidian-canvas-architect/evals.jsonconfig731 B
evals/legacy/obsidian-canvas-architect/results.tsvother301 B
evals/legacy/obsidian-graph-traversal/evals.jsonconfig706 B
evals/legacy/obsidian-graph-traversal/results.tsvother301 B
evals/legacy/obsidian-init/evals.jsonconfig667 B
evals/legacy/obsidian-init/results.tsvother301 B
evals/legacy/obsidian-markdown-mastery/evals.jsonconfig652 B
evals/legacy/obsidian-markdown-mastery/results.tsvother301 B
evals/legacy/obsidian-query-agent/evals.jsonconfig767 B
evals/legacy/obsidian-rlm-distiller/evals.jsonconfig685 B
evals/legacy/obsidian-vault-crud/evals.jsonconfig638 B
evals/legacy/obsidian-vault-crud/results.tsvother301 B
evals/legacy/obsidian-wiki-builder/evals.jsonconfig732 B
evals/legacy/obsidian-wiki-linter/evals.jsonconfig1.5 KB
evals/results.tsvother73 B
evals/task-success.jsonconfig3.5 KB
references/acceptance-criteria.mdmarkdown2.9 KB
references/architecture-background.mdmarkdown2.8 KB
references/dependencies.mdmarkdown1.7 KB
references/interactive-setup.mdmarkdown7.0 KB
references/interactive-stack-setup.mdmarkdown5.6 KB
references/kepano-analysis-summary.mdmarkdown1.7 KB
references/obsidian-bases-manager-acceptance-criteria.mdmarkdown628 B
references/obsidian-bases-manager-fallback-tree.mdmarkdown787 B
references/obsidian-bases-manager.mdmarkdown2.1 KB
references/obsidian-canvas-architect-acceptance-criteria.mdmarkdown645 B
references/obsidian-canvas-architect-fallback-tree.mdmarkdown836 B
references/obsidian-canvas-architect.mdmarkdown1.9 KB
references/obsidian-graph-traversal-acceptance-criteria.mdmarkdown690 B
references/obsidian-graph-traversal-fallback-tree.mdmarkdown877 B
references/obsidian-graph-traversal.mdmarkdown2.2 KB
references/obsidian-init-acceptance-criteria.mdmarkdown650 B
references/obsidian-init-fallback-tree.mdmarkdown1.1 KB
references/obsidian-init.mdmarkdown2.0 KB
references/obsidian-markdown-mastery-acceptance-criteria.mdmarkdown700 B
references/obsidian-markdown-mastery-fallback-tree.mdmarkdown889 B
references/obsidian-markdown-mastery.mdmarkdown2.2 KB
references/obsidian-query-agent.mdmarkdown1.8 KB
references/obsidian-rlm-distiller.mdmarkdown1.8 KB
references/obsidian-vault-crud-acceptance-criteria.mdmarkdown833 B
references/obsidian-vault-crud-fallback-tree.mdmarkdown1.1 KB
references/obsidian-vault-crud.mdmarkdown2.2 KB
references/obsidian-vault-onboarding-guide.mdmarkdown3.7 KB
references/obsidian-wiki-builder.mdmarkdown1.8 KB
references/obsidian-wiki-linter.mdmarkdown1.7 KB
references/resource-index.mdmarkdown4.3 KB
references/safety-learnings.mdmarkdown2.7 KB
references/wiki-builder-guide.mdmarkdown2.7 KB
references/wiki-distillation-guide.mdmarkdown1.7 KB
references/wiki-linting-guide.mdmarkdown2.1 KB
references/wiki-query-guide.mdmarkdown2.1 KB
requirements.inother32 B
requirements.txtmanifest463 B
scripts/audit.pyscript9.0 KB
scripts/bases_ops.pyscript7.8 KB
scripts/canvas_gen.pyscript7.9 KB
scripts/canvas_ops.pyscript11.5 KB
scripts/concept_extractor.pyscript10.2 KB
scripts/constants.pyscript908 B
scripts/distill_wiki.pyscript12.9 KB
scripts/graph_ops.pyscript11.7 KB
scripts/ingest.pyscript8.3 KB
scripts/init_vault.pyscript5.7 KB
scripts/lint_wiki.pyscript9.3 KB
scripts/obsidian-parser/parser.pyscript6.5 KB
scripts/obsidian-parser/tests/test_parser.pyscript4.1 KB
scripts/query_wiki.pyscript21.3 KB
scripts/raw_manifest.pyscript14.3 KB
scripts/vault_ops.pyscript13.4 KB
scripts/wiki_builder.pyscript14.3 KB

scripts/distill_wiki.py

View on GitHub
#!/usr/bin/env python
"""
distill_wiki.py
=====================================

Purpose:
    Cheap-model RLM distillation orchestrator for the Obsidian Wiki Engine.
    Iterates over all wiki nodes needing RLM summaries, auto-detects the
    cheapest available LLM CLI, and generates three summary layers per concept:
    summary.md, bullets.md, deep.md.

    Distillation is performed entirely within this plugin — no cross-plugin
    script calls. RLM summaries are written to a configurable cache directory
    (default: {wiki_root}/rlm/) or to a shared path under .agent/learning/
    when --rlm-cache-dir is set to point there.

    Fallback chain (strict — Ollama is fully deprecated):
        1. copilot  -> gpt-5-mini
        2. claude   -> claude-haiku-4-5
        3. gemini   -> gemini-3-flash-preview

Layer: Distill / Wiki

Usage:
    python ./scripts/distill_wiki.py --wiki-root /path/to/wiki-root
    python ./scripts/distill_wiki.py --wiki-root /path/to/wiki-root --source arch-docs
    python ./scripts/distill_wiki.py --wiki-root /path/to/wiki-root --engine claude
    python ./scripts/distill_wiki.py --wiki-root /path/to/wiki-root --dry-run
    python ./scripts/distill_wiki.py --wiki-root /path/to/wiki-root \\
        --rlm-cache-dir /path/to/project/.agent/learning/rlm_wiki_cache

Related:
    - raw_manifest.py  (WikiSourceConfig + agent-memory.json)
    - wiki_builder.py  (consumes the rlm/ summaries it produces)
    - audit.py         (reports missing summaries)

Key Input Dependencies:
    - Wiki node files under {wiki_root}/wiki/*.md
    - One of copilot/claude/gemini CLIs available on PATH
"""
import sys
import os
import shutil
import argparse
import subprocess
from pathlib import Path
from typing import List, Optional, Tuple

SCRIPT_DIR = Path(__file__).resolve().parent
if str(SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(SCRIPT_DIR))

from raw_manifest import now_iso

# ─── ENGINE REGISTRY (strict cheap-model only) ────────────────────────────────
# Ollama is fully deprecated for distillation tasks.
ENGINE_PRIORITY: List[Tuple[str, str]] = [
    ("copilot", "gpt-5-mini"),
    ("claude",  "claude-haiku-4-5"),
    ("gemini",  "gemini-3-flash-preview"),
]

# ─── PATH AUGMENTATION (stripped PATH in some environments) ──────────────────
_EXTRA_PATHS = [
    "/opt/homebrew/bin",
    "/usr/local/bin",
    os.path.expanduser("~/.local/bin"),
    os.path.expanduser("~/.npm-global/bin"),
    os.path.expanduser("~/n/bin"),
]
for _p in _EXTRA_PATHS:
    if _p not in os.environ.get("PATH", "") and Path(_p).exists():
        os.environ["PATH"] = _p + os.pathsep + os.environ.get("PATH", "")


def detect_engine(override: Optional[str] = None) -> Tuple[str, str]:
    """
    Detect the cheapest available LLM CLI on PATH.

    Args:
        override: Force a specific engine name (e.g. 'claude').

    Returns:
        (engine_name, model_name) tuple.

    Raises:
        SystemExit: If override is unknown or no engine is found.
    """
    if override:
        for engine, model in ENGINE_PRIORITY:
            if engine == override.lower():
                if shutil.which(engine):
                    return (engine, model)
                print(f"[ERROR] Engine '{override}' not found on PATH.")
                print(f"   Install: https://github.com/cli/{override}")
                sys.exit(1)
        print(f"[ERROR] Unknown engine '{override}'. Choose from: {[e for e, _ in ENGINE_PRIORITY]}")
        sys.exit(1)

    for engine, model in ENGINE_PRIORITY:
        if shutil.which(engine):
            print(f"[ENGINE] Using: {engine} ({model})")
            return (engine, model)

    print("[ERROR] No LLM CLI found on PATH.")
    print("   Install one of:")
    for engine, model in ENGINE_PRIORITY:
        print(f"     {engine} (model: {model})")
    sys.exit(1)


def build_prompt(layer: str, concept: str, content: str) -> str:
    """
    Build the distillation prompt for the given layer.

    Args:
        layer:   One of 'summary', 'bullets', 'deep'.
        concept: The concept slug being distilled.
        content: The raw wiki node content excerpt.

    Returns:
        Prompt string ready to send to the LLM.
    """
    base = f"Concept: {concept}\n\n{content}\n\n"
    if layer == "summary":
        return base + (
            "Write a concise 1-5 sentence summary of this concept. "
            "Focus on the most important idea. Plain prose, no headings, no bullets."
        )
    elif layer == "bullets":
        return base + (
            "List 6-10 key ideas from this concept as concise bullet points. "
            "Use '- ' prefix. No headings, no preamble."
        )
    elif layer == "deep":
        return base + (
            "Write a thorough multi-paragraph distillation of this concept. "
            "Cover the main ideas, important details, and relationships to related concepts. "
            "Use markdown headings where helpful."
        )
    return base + "Summarize this content."


def _build_llm_cmd(engine: str, model: str, prompt: str) -> Optional[Tuple[List[str], str]]:
    """Build the (subprocess command, stdin text) pair for the given engine, or None if unknown."""
    if engine == "claude":
        return (["claude", "--model", model, "-p", prompt, "--no-session-persistence"], "")
    if engine == "gemini":
        return (["gemini", "--model", model, "-p", prompt], "")
    if engine == "copilot":
        return (["copilot", "--model", model], prompt)
    return None


def call_llm(
    engine: str,
    model: str,
    prompt: str,
    timeout: int = 120,
) -> Optional[str]:
    """
    Call the specified LLM CLI with the given prompt.

    Args:
        engine:  CLI name ('copilot', 'claude', 'gemini').
        model:   Model identifier string.
        prompt:  Full prompt payload.
        timeout: Subprocess timeout in seconds.

    Returns:
        Stripped LLM response string, or None on error.
    """
    built = _build_llm_cmd(engine, model, prompt)
    if built is None:
        return None
    cmd, stdin_text = built

    try:
        proc = subprocess.run(
            cmd,
            input=stdin_text or None,
            capture_output=True,
            text=True,
            timeout=timeout,
            shell=(sys.platform == "win32"),
        )
    except FileNotFoundError:
        print(f"[ERROR] CLI '{engine}' not found on PATH.")
        return None
    except subprocess.TimeoutExpired:
        print(f"[ERROR] {engine} timed out after {timeout}s.")
        return None

    if proc.returncode != 0 or not proc.stdout.strip():
        print(f"[ERROR] {engine} returned exit code {proc.returncode}")
        if proc.stderr:
            print(proc.stderr[:300])
        return None

    return proc.stdout.strip()


def _resolve_rlm_cache_dir(wiki_root: Path, rlm_cache_dir: Optional[Path] = None) -> Path:
    """
    Resolve the RLM cache directory.

    Priority:
        1. Explicit --rlm-cache-dir argument (e.g. .agent/learning/rlm_wiki_cache)
        2. Default: {wiki_root}/rlm/

    Using --rlm-cache-dir allows the wiki engine to write summaries into the
    same .agent/learning/ tree as rlm-factory, avoiding duplicate storage.
    """
    if rlm_cache_dir:
        return Path(rlm_cache_dir).resolve()
    return wiki_root / "rlm"


def write_rlm_layer(
    concept: str,
    layer: str,
    content: str,
    rlm_cache_dir: Path,
) -> None:
    """
    Write one RLM summary layer file to {rlm_cache_dir}/{concept}/{layer}.md.

    Args:
        concept:       Concept slug.
        layer:         Layer name ('summary', 'bullets', 'deep').
        content:       Distilled text to write.
        rlm_cache_dir: Resolved RLM cache directory.
    """
    rlm_dir = rlm_cache_dir / concept
    rlm_dir.mkdir(parents=True, exist_ok=True)
    layer_path = rlm_dir / f"{layer}.md"

    layer_content = "\n".join([
        "---",
        f'concept: "{concept}"',
        f'layer: "{layer}"',
        f'generated_at: "{now_iso()}"',
        "---",
        "",
        f"# {layer.title()}",
        "",
        content,
        "",
    ])
    layer_path.write_text(layer_content, encoding="utf-8")


def distill_concept(
    concept: str,
    wiki_root: Path,
    engine: str,
    model: str,
    layers: List[str],
    rlm_cache_dir: Path,
    dry_run: bool = False,
) -> bool:
    """
    Distill all requested layers for one concept.

    Reads the wiki node file as source content, calls the LLM for each layer,
    and writes the result to {rlm_cache_dir}/{concept}/

    Args:
        concept:       Concept slug (matches wiki/{concept}.md filename).
        wiki_root:     Root of the wiki output directory.
        engine:        LLM CLI to use.
        model:         Model identifier.
        layers:        List of layer names to generate.
        rlm_cache_dir: Resolved RLM cache directory.
        dry_run:       If True, build prompts but do not call LLM.

    Returns:
        True on success, False if the wiki node file is missing.
    """
    wiki_node = wiki_root / "wiki" / f"{concept}.md"
    if not wiki_node.exists():
        print(f"  [SKIP] Wiki node not found: {wiki_node.name}")
        return False

    content = wiki_node.read_text(encoding="utf-8")

    for layer in layers:
        prompt = build_prompt(layer, concept, content[:3000])

        if dry_run:
            print(f"  [DRY] Would distill {concept}/{layer}.md via {engine}")
            continue

        print(f"  [RUN] {concept}/{layer}.md via {engine}...", end=" ", flush=True)
        result = call_llm(engine, model, prompt)
        if result:
            write_rlm_layer(concept, layer, result, rlm_cache_dir)
            print(f"OK ({len(result)} chars)")
        else:
            print("FAILED")

    return True


def get_concepts_needing_distillation(
    wiki_root: Path,
    layers: List[str],
    rlm_cache_dir: Path,
    source_name: Optional[str] = None,
) -> List[str]:
    """
    Find all wiki node concepts that are missing one or more RLM layers.

    Args:
        wiki_root:     Root of the wiki output directory.
        layers:        Layer names to check for.
        rlm_cache_dir: Resolved RLM cache directory.
        source_name:   Optional source filter (not yet implemented at this level).

    Returns:
        Sorted list of concept slugs needing distillation.
    """
    wiki_dir = wiki_root / "wiki"
    if not wiki_dir.exists():
        return []

    needs_distill = []
    for node_file in sorted(wiki_dir.glob("*.md")):
        if node_file.name.startswith("_"):
            continue
        concept = node_file.stem
        concept_rlm_dir = rlm_cache_dir / concept
        missing = any(not (concept_rlm_dir / f"{layer}.md").exists() for layer in layers)
        if missing:
            needs_distill.append(concept)

    return needs_distill


def main() -> None:
    """Parse CLI arguments and run the distillation pipeline."""
    parser = argparse.ArgumentParser(
        description="Distill wiki nodes into RLM summary layers using cheap cloud LLMs"
    )
    parser.add_argument("--wiki-root", required=True, help="Path to the wiki root directory")
    parser.add_argument("--source", default=None, help="Distill one named source only (filter by source label)")
    parser.add_argument("--engine", default=None, choices=["copilot", "claude", "gemini"],
                        help="Force a specific LLM CLI engine")
    parser.add_argument("--layers", default="summary,bullets,deep",
                        help="Comma-separated layers to generate (default: summary,bullets,deep)")
    parser.add_argument(
        "--rlm-cache-dir",
        default=None,
        help=(
            "Override the RLM cache directory (default: {wiki-root}/rlm). "
            "Set to .agent/learning/rlm_wiki_cache to colocate with rlm-factory."
        ),
    )
    parser.add_argument("--dry-run", action="store_true",
                        help="Show what would be distilled without calling the LLM")
    args = parser.parse_args()

    wiki_root = Path(args.wiki_root).resolve()
    layers = [layer.strip() for layer in args.layers.split(",")]
    engine, model = detect_engine(args.engine)
    rlm_cache_dir = _resolve_rlm_cache_dir(wiki_root, args.rlm_cache_dir)

    concepts = get_concepts_needing_distillation(wiki_root, layers, rlm_cache_dir)
    if not concepts:
        print("[OK] All wiki nodes already have RLM summaries.")
        return

    print(f"\n[DISTILL] {len(concepts)} concepts need distillation")
    print(f"          Engine   : {engine} ({model})")
    print(f"          Layers   : {layers}")
    print(f"          Wiki     : {wiki_root}")
    print(f"          RLM cache: {rlm_cache_dir}\n")

    success = 0
    for concept in concepts:
        ok = distill_concept(
            concept, wiki_root, engine, model, layers, rlm_cache_dir,
            dry_run=args.dry_run,
        )
        if ok:
            success += 1

    action = "planned" if args.dry_run else "completed"
    print(f"\n[DONE] Distillation {action}: {success}/{len(concepts)} concepts.")


if __name__ == "__main__":
    main()
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall