Skip to content

richfrem/agent-plugins-skills / plugins/agent-memory/.apm/skills/managing-obsidian-vaults

managing-obsidian-vaults

Verified

Interactive vault, wiki and Super-RAG setup in the main conversation. Use for "set up my wiki", "initialize the full knowledge stack" or combined RLM/vector/wiki setup. Manage Obsidian vaults and LLM wikis: initialize vaults, author Markdown and wikilinks, safely edit notes and Bases, create Canvas diagrams, traverse graphs, build concept wikis, distill summary/bullets/deep layers, query and lint knowledge.

FileKindSize
SKILL.mdskill md5.9 KB
assets/templates/config.yaml.templateother556 B
assets/templates/raw-sources.json.exampleother862 B
assets/templates/wiki-node.template.mdmarkdown410 B
evals/README.mdmarkdown569 B
evals/evals.jsonconfig23.4 KB
evals/legacy/obsidian-bases-manager/evals.jsonconfig692 B
evals/legacy/obsidian-bases-manager/results.tsvother301 B
evals/legacy/obsidian-canvas-architect/evals.jsonconfig731 B
evals/legacy/obsidian-canvas-architect/results.tsvother301 B
evals/legacy/obsidian-graph-traversal/evals.jsonconfig706 B
evals/legacy/obsidian-graph-traversal/results.tsvother301 B
evals/legacy/obsidian-init/evals.jsonconfig667 B
evals/legacy/obsidian-init/results.tsvother301 B
evals/legacy/obsidian-markdown-mastery/evals.jsonconfig652 B
evals/legacy/obsidian-markdown-mastery/results.tsvother301 B
evals/legacy/obsidian-query-agent/evals.jsonconfig767 B
evals/legacy/obsidian-rlm-distiller/evals.jsonconfig685 B
evals/legacy/obsidian-vault-crud/evals.jsonconfig638 B
evals/legacy/obsidian-vault-crud/results.tsvother301 B
evals/legacy/obsidian-wiki-builder/evals.jsonconfig732 B
evals/legacy/obsidian-wiki-linter/evals.jsonconfig1.5 KB
evals/results.tsvother73 B
evals/task-success.jsonconfig3.5 KB
references/acceptance-criteria.mdmarkdown2.9 KB
references/architecture-background.mdmarkdown2.8 KB
references/dependencies.mdmarkdown1.7 KB
references/interactive-setup.mdmarkdown7.0 KB
references/interactive-stack-setup.mdmarkdown5.6 KB
references/kepano-analysis-summary.mdmarkdown1.7 KB
references/obsidian-bases-manager-acceptance-criteria.mdmarkdown628 B
references/obsidian-bases-manager-fallback-tree.mdmarkdown787 B
references/obsidian-bases-manager.mdmarkdown2.1 KB
references/obsidian-canvas-architect-acceptance-criteria.mdmarkdown645 B
references/obsidian-canvas-architect-fallback-tree.mdmarkdown836 B
references/obsidian-canvas-architect.mdmarkdown1.9 KB
references/obsidian-graph-traversal-acceptance-criteria.mdmarkdown690 B
references/obsidian-graph-traversal-fallback-tree.mdmarkdown877 B
references/obsidian-graph-traversal.mdmarkdown2.2 KB
references/obsidian-init-acceptance-criteria.mdmarkdown650 B
references/obsidian-init-fallback-tree.mdmarkdown1.1 KB
references/obsidian-init.mdmarkdown2.0 KB
references/obsidian-markdown-mastery-acceptance-criteria.mdmarkdown700 B
references/obsidian-markdown-mastery-fallback-tree.mdmarkdown889 B
references/obsidian-markdown-mastery.mdmarkdown2.2 KB
references/obsidian-query-agent.mdmarkdown1.8 KB
references/obsidian-rlm-distiller.mdmarkdown1.8 KB
references/obsidian-vault-crud-acceptance-criteria.mdmarkdown833 B
references/obsidian-vault-crud-fallback-tree.mdmarkdown1.1 KB
references/obsidian-vault-crud.mdmarkdown2.2 KB
references/obsidian-vault-onboarding-guide.mdmarkdown3.7 KB
references/obsidian-wiki-builder.mdmarkdown1.8 KB
references/obsidian-wiki-linter.mdmarkdown1.7 KB
references/resource-index.mdmarkdown4.3 KB
references/safety-learnings.mdmarkdown2.7 KB
references/wiki-builder-guide.mdmarkdown2.7 KB
references/wiki-distillation-guide.mdmarkdown1.7 KB
references/wiki-linting-guide.mdmarkdown2.1 KB
references/wiki-query-guide.mdmarkdown2.1 KB
requirements.inother32 B
requirements.txtmanifest463 B
scripts/audit.pyscript9.0 KB
scripts/bases_ops.pyscript7.8 KB
scripts/canvas_gen.pyscript7.9 KB
scripts/canvas_ops.pyscript11.5 KB
scripts/concept_extractor.pyscript10.2 KB
scripts/constants.pyscript908 B
scripts/distill_wiki.pyscript12.9 KB
scripts/graph_ops.pyscript11.7 KB
scripts/ingest.pyscript8.3 KB
scripts/init_vault.pyscript5.7 KB
scripts/lint_wiki.pyscript9.3 KB
scripts/obsidian-parser/parser.pyscript6.5 KB
scripts/obsidian-parser/tests/test_parser.pyscript4.1 KB
scripts/query_wiki.pyscript21.3 KB
scripts/raw_manifest.pyscript14.3 KB
scripts/vault_ops.pyscript13.4 KB
scripts/wiki_builder.pyscript14.3 KB

scripts/concept_extractor.py

View on GitHub
#!/usr/bin/env python
"""
concept_extractor.py
=====================================

Purpose:
    Cross-source concept synthesis for the Obsidian Wiki Engine.
    Implements the "compile" step from Karpathy's LLM wiki vision:
    multiple raw source files about the same concept are merged into
    one authoritative wiki node instead of producing N separate pages.

    Reads ParsedRecord JSON from stdin (or --input file), groups records by
    concept slug, merges multi-source records into a single synthetic record,
    improves cluster assignment using keyword extraction, and writes the
    deduplicated record list to stdout (or --output file).

    Called by wiki_builder.py between ingest.py and node generation.

Layer: Extract / Wiki

Usage:
    # Pipe from ingest.py
    python ingest.py --wiki-root /path/to/wiki-root | python concept_extractor.py --json

    # From file
    python concept_extractor.py --input /tmp/records.json --output /tmp/merged.json

Related:
    - ingest.py       (produces ParsedRecord input)
    - wiki_builder.py (consumes merged output)

Key Input Dependencies:
    - ParsedRecord JSON from ingest.py, via stdin or --input file
"""
import sys
import json
import re
import argparse
from pathlib import Path
from typing import Dict, List, Any

SCRIPT_DIR = Path(__file__).resolve().parent
if str(SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(SCRIPT_DIR))

# ─── STOPWORDS ────────────────────────────────────────────────────────────────
# Common English words that don't carry cluster-level meaning
_STOPWORDS = frozenset({
    "a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
    "of", "with", "by", "from", "as", "is", "are", "was", "were", "be",
    "been", "being", "have", "has", "had", "do", "does", "did", "will",
    "would", "could", "should", "may", "might", "shall", "can", "this",
    "that", "these", "those", "it", "its", "i", "we", "you", "they",
    "he", "she", "not", "no", "all", "also", "more", "over", "into",
    "about", "up", "out", "if", "then", "than", "so", "yet", "both",
    "each", "when", "there", "here", "how", "what", "which", "who",
    "any", "one", "two", "new", "use", "used", "using", "s",
})


def _extract_keywords(text: str, top_n: int = 3) -> List[str]:
    """
    Extract the top N keyword tokens from text by term frequency.

    Strips markdown syntax, splits on whitespace/punctuation, removes
    stopwords, and returns the most frequent remaining tokens.

    Args:
        text:  Raw content string (markdown).
        top_n: Number of top keywords to return.

    Returns:
        List of keyword strings (lowercase, deduplicated, frequency-sorted).
    """
    # Strip markdown: remove code fences, links, wikilinks, headings syntax
    text = re.sub(r"```[\s\S]*?```", " ", text)
    text = re.sub(r"`[^`]+`", " ", text)
    text = re.sub(r"\[\[([^\]]+)\]\]", r"\1", text)
    text = re.sub(r"\[([^\]]+)\]\([^)]+\)", r"\1", text)
    text = re.sub(r"#{1,6}\s*", " ", text)
    text = re.sub(r"[^\w\s]", " ", text.lower())

    tokens = text.split()
    freq: Dict[str, int] = {}
    for tok in tokens:
        if len(tok) >= 4 and tok not in _STOPWORDS:
            freq[tok] = freq.get(tok, 0) + 1

    sorted_tokens = sorted(freq.keys(), key=lambda t: -freq[t])
    return sorted_tokens[:top_n]


def infer_cluster_from_content(source_label: str, concept: str, content: str) -> str:
    """
    Infer a semantic cluster name from content keywords.

    Uses top keyword(s) extracted from the content as the cluster name,
    falling back to source_label if extraction yields nothing useful.

    The cluster name is kebab-case and represents the dominant topic of
    the concept, not just its source origin.

    Args:
        source_label: The source label (used as fallback cluster).
        concept:      The concept slug.
        content:      Raw source content for keyword extraction.

    Returns:
        Kebab-case cluster name.
    """
    keywords = _extract_keywords(content, top_n=2)

    # If top keyword is too similar to concept itself, use the second
    concept_tokens = set(concept.split("-"))
    filtered = [k for k in keywords if k not in concept_tokens]
    if filtered:
        return filtered[0]

    # Use source label if no distinguishing keyword found
    return source_label


def _combine_record_contents(records_sorted: List[Dict[str, Any]]) -> str:
    """Concatenate content from all sorted records with source attribution headers, truncated if too long."""
    combined_parts = []
    for r in records_sorted:
        label = r.get("source_label", r.get("source_name", "unknown"))
        src_file = r.get("source_file", "")
        combined_parts.append(f"<!-- Source: {label}/{src_file} -->\n{r.get('content', '')}")

    MAX_COMBINED = 5000
    combined_content = "\n\n".join(combined_parts)
    if len(combined_content) > MAX_COMBINED:
        combined_content = combined_content[:MAX_COMBINED] + "\n\n*(combined content truncated)*"
    return combined_content


def _merge_records(records: List[Dict[str, Any]]) -> Dict[str, Any]:
    """
    Merge multiple ParsedRecords for the same concept into one authoritative record.

    The merged record:
    - Keeps the title from the first (longest) record
    - Combines content from all sources (truncated per-source)
    - Lists all source files in 'source_files' (new multi-source field)
    - Sets 'source_label' to the primary source, 'source_labels' to all
    - Re-runs keyword extraction on combined content for cluster assignment
    - Preserves the latest 'generated_at' timestamp

    Args:
        records: List of ParsedRecord dicts sharing the same concept slug.

    Returns:
        Single merged ParsedRecord dict.
    """
    if len(records) == 1:
        return records[0]

    # Sort: put the record with most content first (primary source)
    records_sorted = sorted(records, key=lambda r: len(r.get("content", "")), reverse=True)
    primary = records_sorted[0]

    # Combine content with source attribution headers
    combined_content = _combine_record_contents(records_sorted)

    # Re-infer cluster from combined content (more representative)
    cluster = infer_cluster_from_content(
        primary.get("source_label", ""),
        primary.get("concept", ""),
        combined_content,
    )

    merged: Dict[str, Any] = {
        **primary,
        "content":       combined_content,
        "cluster":       cluster,
        "source_label":  primary.get("source_label", ""),
        "source_labels": [r.get("source_label", "") for r in records_sorted],
        "source_file":   primary.get("source_file", ""),
        "source_files":  [r.get("source_file", "") for r in records_sorted],
        "source_names":  [r.get("source_name", "") for r in records_sorted],
        "generated_at":  max(r.get("generated_at", "") for r in records_sorted),
        "multi_source":  True,
    }
    return merged


def extract_concepts(records: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
    """
    Group records by concept slug, merge multi-source groups, improve clusters.

    This is the "compile" step: N source files → M concept nodes (M ≤ N).
    Files from different sources that share the same concept slug are merged
    into one authoritative wiki node with all source attributions preserved.

    Args:
        records: List of ParsedRecord dicts from ingest.py.

    Returns:
        Deduplicated, merged list of ParsedRecord dicts ready for wiki_builder.
    """
    # Group by concept slug
    groups: Dict[str, List[Dict[str, Any]]] = {}
    for record in records:
        concept = record.get("concept", "")
        if not concept:
            continue
        groups.setdefault(concept, []).append(record)

    # Merge each group + improve cluster assignment
    merged_records: List[Dict[str, Any]] = []
    for concept, group in sorted(groups.items()):
        merged = _merge_records(group)

        # Improve cluster for single-source records too
        if not merged.get("multi_source"):
            merged["cluster"] = infer_cluster_from_content(
                merged.get("source_label", ""),
                concept,
                merged.get("content", ""),
            )

        merged_records.append(merged)

    return merged_records


def main() -> None:
    """Parse CLI args: read records from stdin/file, write merged records to stdout/file."""
    parser = argparse.ArgumentParser(
        description="Merge multi-source ParsedRecords by concept slug (cross-source synthesis)"
    )
    parser.add_argument("--input", default=None, help="Input records JSON file (default: stdin)")
    parser.add_argument("--output", default=None, help="Output merged records JSON file (default: stdout)")
    parser.add_argument("--json", dest="json_output", action="store_true",
                        help="Force JSON output (default when --output not set)")
    parser.add_argument("--stats", action="store_true",
                        help="Print summary stats instead of full JSON")
    args = parser.parse_args()

    # Read input
    if args.input:
        raw_text = Path(args.input).read_text(encoding="utf-8")
    else:
        raw_text = sys.stdin.read()

    try:
        records: List[Dict[str, Any]] = json.loads(raw_text)
    except json.JSONDecodeError as exc:
        print(f"[ERROR] Could not parse input JSON: {exc}", file=sys.stderr)
        sys.exit(1)

    merged = extract_concepts(records)

    if args.stats:
        multi = [r for r in merged if r.get("multi_source")]
        print(f"[EXTRACT] Input records : {len(records)}")
        print(f"[EXTRACT] Output concepts: {len(merged)}")
        print(f"[EXTRACT] Merged nodes  : {len(multi)}")
        clusters = {r["cluster"] for r in merged}
        print(f"[EXTRACT] Clusters found: {len(clusters)}: {sorted(clusters)}")
        return

    result_json = json.dumps(merged, indent=2, ensure_ascii=False)

    if args.output:
        out_path = Path(args.output)
        out_path.parent.mkdir(parents=True, exist_ok=True)
        out_path.write_text(result_json, encoding="utf-8")
        print(f"[EXTRACT] {len(records)} records → {len(merged)} concepts written: {args.output}",
              file=sys.stderr)
    else:
        print(result_json)


if __name__ == "__main__":
    main()
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall