Skip to content

praneybehl/llm-wiki-plugin / skills/llm-wiki

llm-wiki

Verified

Build and maintain an LLM-curated knowledge base from papers, articles, transcripts, notes and project findings. Ingest sources into linked Markdown, answer questions with citations, lint or upgrade a wiki, and preserve useful synthesis across sessions and agents. Also capture verified task experience, consolidate success/failure patterns, and propose, evaluate, apply or roll back evidence-linked skill improvements. Trigger on "add this to my wiki", "what does the wiki say", "learn from this task", "improve this procedure", /wiki:learn or /wiki:evolve. Works on a local folder or a hosted wiki URL (MCP). Use index-first navigation, atomic pages and hybrid search.

FileKindSize
SKILL.mdskill md24.6 KB
assets/SCHEMA.md.templateother7.4 KB
assets/decision-questions.jsonconfig3.1 KB
assets/evolution_README.md.templateother2.3 KB
assets/experience.json.templateother660 B
assets/graph_README.md.templateother2.0 KB
assets/graph_gitignore.templateother27 B
assets/index.md.templateother775 B
assets/log.md.templateother499 B
assets/ontology.yaml.templateother3.2 KB
assets/page.md.templateother805 B
assets/pattern.md.templateother839 B
assets/wiki-cache_gitignore.templateother14 B
references/agent-memory-integration.mdmarkdown7.5 KB
references/architecture.mdmarkdown9.6 KB
references/decision-models.mdmarkdown6.5 KB
references/evolution-workflow.mdmarkdown19.7 KB
references/graph-workflow.mdmarkdown7.1 KB
references/hosted-mode.mdmarkdown11.4 KB
references/ingest-workflow.mdmarkdown11.1 KB
references/lint-workflow.mdmarkdown8.3 KB
references/page-conventions.mdmarkdown5.8 KB
references/query-workflow.mdmarkdown8.6 KB
references/retrieval-setup.mdmarkdown5.6 KB
references/scaling-playbook.mdmarkdown6.8 KB
scripts/init_wiki.pyscript9.5 KB
scripts/setup_wiki.pyexecutablescript2.5 KB
scripts/test_link_resolution.pyscript6.2 KB
scripts/wiki_decide.pyexecutablescript31.6 KB
scripts/wiki_evolve.pyscript29.1 KB
scripts/wiki_evolve_agent.pyscript15.1 KB
scripts/wiki_evolve_api.pyscript7.0 KB
scripts/wiki_evolve_claude.pyscript2.4 KB
scripts/wiki_evolve_loop.pyscript43.4 KB
scripts/wiki_evolve_stream.pyscript13.0 KB
scripts/wiki_graph_extract.pyscript21.0 KB
scripts/wiki_graph_lint.pyscript22.0 KB
scripts/wiki_graph_query.pyscript9.3 KB
scripts/wiki_lint.pyscript14.5 KB
scripts/wiki_remote.pyexecutablescript76.6 KB
scripts/wiki_search.pyscript35.9 KB
scripts/wiki_stats.pyscript5.9 KB

scripts/wiki_lint.py

View on GitHub
#!/usr/bin/env python3
"""
wiki_lint.py — Structural health check for an LLM Wiki.

Reports orphan pages, broken wikilinks, oversized pages, frontmatter issues,
stale pages, duplicate slugs, and (with --suggest-pages) terms that appear in
many pages without their own page.

Conservative by design: reports findings, never edits.

Usage:
    python wiki_lint.py [<wiki-dir>] [options]

Options:
    --soft-cap N        Page-size soft cap in lines (default: 400)
    --hard-cap N        Page-size hard cap in lines (default: 800)
    --required-fm a,b   Required frontmatter fields (default: type,title,tags,created,updated)
    --suggest-pages     Surface terms appearing in many pages without a page
    --suggest-min N     Minimum occurrences for --suggest-pages (default: 5)
    --json              Emit JSON instead of text

Examples:
    python wiki_lint.py wiki/
    python wiki_lint.py wiki/ --suggest-pages
    python wiki_lint.py wiki/ --json > lint.json
"""

import argparse
import json
import re
import sys
from collections import Counter, defaultdict
from datetime import date, datetime
from pathlib import Path


WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
CAPITALIZED_PHRASE_RE = re.compile(r"\b([A-Z][a-zA-Z0-9]+(?:\s+[A-Z][a-zA-Z0-9]+){0,3})\b")


def clean_link_target(link: str) -> str:
    """Strip a wikilink down to its path, dropping alias, anchor, and .md."""
    target = link.split("|", 1)[0]      # drop a display alias, if any
    target = target.split("#", 1)[0]    # drop a heading anchor
    target = target.replace("\\", "/").strip().strip("/")
    if target.lower().endswith(".md"):
        target = target[:-3]
    return target.strip()


def resolve_link(link: str, by_path: dict, by_slug) -> "str | None":
    """Resolve a wikilink to the slug of the page it names, or None if broken.

    Wikilinks appear both bare ([[kalman-filter]]) and path-qualified
    ([[entities/kalman-filter]]), optionally with a heading anchor or an
    explicit .md. A page's identity is its filename stem, so comparing raw
    link text misses every path-qualified form — reporting it broken and
    contributing no inbound edge, which then also misreports well-referenced
    pages as orphans.

    A directory prefix is treated as a CONSTRAINT, not decoration: a qualified
    link resolves only if that exact path exists. Falling back to a bare-stem
    match would let [[raw/foo]] silently bind to sources/foo.md — a live
    collision pattern wherever source pages are named after their raw files —
    turning a genuinely broken out-of-tree link into a false pass. Unresolvable
    is the safe answer; a wrong resolution is not.

    Matching is case-SENSITIVE in both forms. Case-folding paths (but not bare
    slugs) would be an asymmetry, and of the two ways to be wrong, a spurious
    "broken link" report is visible and harmless while a spurious resolution is
    silent.
    """
    target = clean_link_target(link)
    if not target:
        return None
    if "/" in target:
        return by_path.get(target)
    return target if target in by_slug else None


SKIP_TOP_LEVEL_FILES = {"SCHEMA.md", "index.md", "log.md", "README.md"}
SKIP_TOP_LEVEL_DIRS = {"indexes", "graph", "raw", ".evolution"}


def parse_frontmatter(text: str) -> tuple[dict, str, bool]:
    """Returns (metadata, body, malformed). malformed=True if frontmatter was attempted but unparseable."""
    if not text.startswith("---"):
        return {}, text, False
    m = FRONTMATTER_RE.match(text)
    if not m:
        return {}, text, True
    fm_text = m.group(1)
    body = text[m.end():]
    meta = {}
    current_key = None
    for line in fm_text.split("\n"):
        if not line.strip():
            continue
        kv = re.match(r"^([a-zA-Z_]+):\s*(.*)$", line)
        if kv:
            key, value = kv.group(1), kv.group(2).strip()
            if value.startswith("[") and value.endswith("]"):
                items = [x.strip().strip('"').strip("'") for x in value[1:-1].split(",") if x.strip()]
                meta[key] = items
            elif value:
                meta[key] = value.strip('"').strip("'")
            else:
                meta[key] = []
                current_key = key
        elif line.startswith("  - ") and current_key:
            meta[current_key].append(line[4:].strip().strip('"').strip("'"))
    return meta, body, False


def collect_pages(wiki_root: Path) -> list[dict]:
    pages = []
    for md_path in wiki_root.rglob("*.md"):
        rel = md_path.relative_to(wiki_root)
        if rel.parts[0] in SKIP_TOP_LEVEL_FILES or rel.parts[0] in SKIP_TOP_LEVEL_DIRS:
            continue
        if rel.name.startswith("."):
            continue
        try:
            text = md_path.read_text(encoding="utf-8")
        except (UnicodeDecodeError, OSError) as e:
            pages.append({
                "path": str(md_path),
                "rel_path": str(rel),
                "slug": md_path.stem,
                "read_error": str(e),
            })
            continue
        meta, body, malformed = parse_frontmatter(text)
        line_count = text.count("\n") + 1
        links = [m.group(1).strip() for m in WIKILINK_RE.finditer(body)]
        pages.append({
            "path": str(md_path),
            "rel_path": str(rel),
            "slug": md_path.stem,
            "meta": meta,
            "body": body,
            "line_count": line_count,
            "links": links,
            "malformed_fm": malformed,
        })
    return pages


def parse_date(s):
    if not s or not isinstance(s, str):
        return None
    try:
        return datetime.strptime(s[:10], "%Y-%m-%d").date()
    except (ValueError, TypeError):
        return None


def lint(pages: list[dict], soft_cap: int, hard_cap: int, required_fm: list[str], suggest_pages: bool, suggest_min: int) -> dict:
    findings = {
        "orphans": [],
        "broken_links": [],
        "oversized_hard": [],
        "oversized_soft": [],
        "missing_frontmatter": [],
        "malformed_frontmatter": [],
        "duplicate_slugs": [],
        "stale_pages": [],
        "read_errors": [],
        "suggested_pages": [],
        "summary": {},
    }

    # Read errors
    for p in pages:
        if "read_error" in p:
            findings["read_errors"].append({"path": p["rel_path"], "error": p["read_error"]})

    pages = [p for p in pages if "read_error" not in p]

    # Slugs
    slug_to_pages = defaultdict(list)
    for p in pages:
        slug_to_pages[p["slug"]].append(p["rel_path"])
    for slug, paths in slug_to_pages.items():
        if len(paths) > 1:
            findings["duplicate_slugs"].append({"slug": slug, "paths": paths})

    # Inbound link map
    inbound = defaultdict(set)
    all_slugs = set(slug_to_pages.keys())
    # Resolution indexes: by full wiki-relative path (sans .md) for
    # path-qualified links, and by bare stem for unqualified ones.
    by_slug = all_slugs
    by_path = {}
    for p in pages:
        rel = p["rel_path"].replace("\\", "/")
        if rel.lower().endswith(".md"):
            rel = rel[:-3]
        by_path[rel] = p["slug"]

    for p in pages:
        for link in p["links"]:
            resolved = resolve_link(link, by_path, by_slug)
            if resolved:
                inbound[resolved].add(p["slug"])

    # Orphans, broken links, oversize, frontmatter, staleness
    for p in pages:
        # Orphans
        if not inbound.get(p["slug"]):
            findings["orphans"].append({"slug": p["slug"], "path": p["rel_path"]})

        # Broken links
        for link in p["links"]:
            if resolve_link(link, by_path, by_slug) is None:
                findings["broken_links"].append({
                    "from": p["slug"],
                    "from_path": p["rel_path"],
                    "to": link,
                })

        # Oversize
        if p["line_count"] > hard_cap:
            findings["oversized_hard"].append({"path": p["rel_path"], "lines": p["line_count"]})
        elif p["line_count"] > soft_cap:
            findings["oversized_soft"].append({"path": p["rel_path"], "lines": p["line_count"]})

        # Frontmatter
        if p["malformed_fm"]:
            findings["malformed_frontmatter"].append({"path": p["rel_path"]})
        else:
            missing = [field for field in required_fm if field not in p["meta"] or p["meta"].get(field) in ("", None, [])]
            if missing:
                findings["missing_frontmatter"].append({"path": p["rel_path"], "missing": missing})

        # Staleness: heuristic — page hasn't been updated in 90 days AND has been touched by recent ingests.
        # Approximate: if updated > 90d ago and the page is well-linked (a hub), flag it.
        updated = parse_date(p["meta"].get("updated"))
        if updated:
            age_days = (date.today() - updated).days
            if age_days > 90 and len(inbound.get(p["slug"], [])) >= 3:
                findings["stale_pages"].append({
                    "path": p["rel_path"],
                    "updated": p["meta"].get("updated"),
                    "age_days": age_days,
                    "inbound_count": len(inbound.get(p["slug"], [])),
                })

    # Suggested pages: capitalized multi-word phrases appearing in many pages without a page
    if suggest_pages:
        phrase_pages = defaultdict(set)
        for p in pages:
            seen = set()
            for m in CAPITALIZED_PHRASE_RE.finditer(p["body"]):
                phrase = m.group(1).strip()
                seen.add(phrase)
            for phrase in seen:
                phrase_pages[phrase].add(p["slug"])

        # Title set for filtering
        existing_titles = {p["meta"].get("title", "").lower() for p in pages}
        existing_slugs_normalized = {s.lower().replace("-", " ") for s in all_slugs}

        candidates = []
        for phrase, page_set in phrase_pages.items():
            if len(page_set) < suggest_min:
                continue
            if phrase.lower() in existing_titles:
                continue
            if phrase.lower() in existing_slugs_normalized:
                continue
            # Filter out section header garbage
            if phrase.split()[0] in {"Section", "Where", "Sources", "Tags", "Type", "Title"}:
                continue
            candidates.append({"phrase": phrase, "page_count": len(page_set), "pages": sorted(page_set)[:5]})
        candidates.sort(key=lambda x: -x["page_count"])
        findings["suggested_pages"] = candidates[:30]

    findings["summary"] = {
        "total_pages": len(pages),
        "orphans": len(findings["orphans"]),
        "broken_links": len(findings["broken_links"]),
        "oversized_hard": len(findings["oversized_hard"]),
        "oversized_soft": len(findings["oversized_soft"]),
        "missing_frontmatter": len(findings["missing_frontmatter"]),
        "malformed_frontmatter": len(findings["malformed_frontmatter"]),
        "duplicate_slugs": len(findings["duplicate_slugs"]),
        "stale_pages": len(findings["stale_pages"]),
        "read_errors": len(findings["read_errors"]),
        "suggested_pages": len(findings["suggested_pages"]),
    }
    return findings


def render_text(findings: dict) -> str:
    out = []
    s = findings["summary"]
    out.append("=" * 60)
    out.append("Wiki Lint Report")
    out.append("=" * 60)
    out.append(f"Total pages scanned: {s['total_pages']}")
    out.append("")

    sections = [
        ("orphans", "Orphan pages (no inbound links)", lambda f: f"  - {f['slug']}  ({f['path']})"),
        ("broken_links", "Broken wikilinks", lambda f: f"  - [[{f['to']}]] referenced from {f['from_path']}"),
        ("oversized_hard", "OVERSIZE (over hard cap — must split)", lambda f: f"  - {f['path']}  ({f['lines']} lines)"),
        ("oversized_soft", "Oversize (over soft cap — consider splitting)", lambda f: f"  - {f['path']}  ({f['lines']} lines)"),
        ("missing_frontmatter", "Missing frontmatter fields", lambda f: f"  - {f['path']}  missing: {', '.join(f['missing'])}"),
        ("malformed_frontmatter", "Malformed frontmatter", lambda f: f"  - {f['path']}"),
        ("duplicate_slugs", "Duplicate slugs", lambda f: f"  - {f['slug']}: {', '.join(f['paths'])}"),
        ("stale_pages", "Stale pages (well-linked but not updated in 90+ days)", lambda f: f"  - {f['path']}  (updated {f['updated']}, {f['age_days']}d ago, {f['inbound_count']} inbound)"),
        ("read_errors", "Read errors", lambda f: f"  - {f['path']}: {f['error']}"),
    ]

    for key, label, formatter in sections:
        items = findings[key]
        if not items:
            continue
        out.append(f"{label} ({len(items)}):")
        for item in items[:50]:
            out.append(formatter(item))
        if len(items) > 50:
            out.append(f"  ... and {len(items) - 50} more")
        out.append("")

    if findings["suggested_pages"]:
        out.append(f"Suggested page candidates ({len(findings['suggested_pages'])}):")
        out.append("  Phrases appearing in many pages without a dedicated page:")
        for item in findings["suggested_pages"]:
            out.append(f"  - \"{item['phrase']}\"  ({item['page_count']} pages)")
        out.append("")

    if all(v == 0 for k, v in s.items() if k != "total_pages"):
        out.append("No issues found. Wiki is healthy.")

    return "\n".join(out)


def main():
    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument("wiki", nargs="?", type=Path, default=Path("wiki"), help="Wiki directory (default: ./wiki).")
    parser.add_argument("--soft-cap", type=int, default=400, help="Page-size soft cap (lines).")
    parser.add_argument("--hard-cap", type=int, default=800, help="Page-size hard cap (lines).")
    parser.add_argument("--required-fm", default="type,title,tags,created,updated", help="Required frontmatter fields, comma-separated.")
    parser.add_argument("--suggest-pages", action="store_true", help="Surface page candidates.")
    parser.add_argument("--suggest-min", type=int, default=5, help="Minimum page count for suggestions.")
    parser.add_argument("--json", action="store_true", help="Emit JSON.")
    args = parser.parse_args()

    if not args.wiki.exists():
        print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
        sys.exit(1)

    pages = collect_pages(args.wiki)
    required_fm = [f.strip() for f in args.required_fm.split(",") if f.strip()]
    findings = lint(pages, args.soft_cap, args.hard_cap, required_fm, args.suggest_pages, args.suggest_min)

    if args.json:
        print(json.dumps(findings, indent=2, default=str))
    else:
        print(render_text(findings))


if __name__ == "__main__":
    main()
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall