Skip to content

po4yka/llm-wiki-skills / skills/llm-wiki-trust-audit

llm-wiki-trust-audit

Verified

Audit an LLM-Wiki system for anti-slop controls such as provenance, review gates, protected human synthesis, stale claims, and unsafe agent write permissions. Use when the user asks whether their wiki can be trusted; route setup/permission safety reviews to llm-wiki-security-review and proposal risk audits to llm-wiki-critique-audit.

LLM-Wiki Trust Audit

Goal

Evaluate whether an LLM-Wiki is trustworthy enough to use for decisions and identify the highest-risk failure modes.

When to use

  • The user asks whether their LLM-Wiki can be trusted before relying on it for a decision.
  • Before granting an agent write access to a vault, or after granting it, to confirm the write-safety boundary still holds.
  • After a large ingest, bulk edit, or lint run, to confirm provenance and review gates were not weakened.
  • Route setup/permission-only reviews to llm-wiki-security-review and proposal-specific risk reviews to llm-wiki-critique-audit instead of this skill.

Inputs

  • Vault/repository path.
  • AGENTS.md, CLAUDE.md, skills and schemas.
  • raw/, wiki/, _meta/, _agent/reports/ if present.
  • Git history if available.
  • User's risk tolerance and domain sensitivity.

Procedure

1. Inspect the trust model

Check whether the system defines:

  • raw source immutability;
  • page lifecycle states;
  • source backlinks;
  • claim types;
  • confidence semantics;
  • review requirements;
  • protected human sections;
  • lint cadence;
  • rollback/recovery path.

2. Audit provenance

Sample important pages and report:

  • no provenance;
  • source-level provenance only;
  • claim-level provenance;
  • generated pages citing generated pages;
  • stale or missing source hashes;
  • source links that no longer resolve.

Re-verify any page flagged stale or with an unresolved source link before treating its claims as trustworthy.

3. Audit generated content boundaries

Look for:

  • AI-generated pages marked reviewed/verified without evidence;
  • ai_confidence defaulted lazily;
  • low-confidence pages not requiring review;
  • ambiguous claims in trusted pages;
  • human synthesis sections overwritten or unprotected.

4. Audit structural health

Coordinate with wiki-lint if available. Check:

  • broken links;
  • orphan pages;
  • duplicate concepts;
  • taxonomy drift;
  • stale pages;
  • contradiction reports ignored;
  • excessive draft backlog.

5. Audit write safety

Check whether agents can:

  • edit raw/;
  • delete source material;
  • bulk rewrite wiki pages without dry-run;
  • bypass git/PR review without explicit approval;
  • follow prompt-injection instructions from captured content;
  • expose sensitive material to external tools.

6. Grade risk

Use this scorecard:

AreaRiskEvidenceFix
Raw immutabilitylow/medium/high
Provenancelow/medium/high
Review gateslow/medium/high
Human synthesis boundarylow/medium/high
Stalenesslow/medium/high
Agent write safetylow/medium/high
Sensitive datalow/medium/high

7. Recommend fixes

Prioritize fixes that reduce trust risk before productivity features:

  1. protect raw sources;
  2. add review states;
  3. add provenance fields;
  4. protect human sections;
  5. add lint reports;
  6. add dry-run/PR workflow;
  7. add retrieval upgrades only after trust basics.

Output

## Trust audit summary

## Top risks

## Evidence

## Scorecard

## Fix plan

## Stop conditions

## Optional follow-up skills

Safety gates

  • Do not mark the wiki safe without evidence.
  • Do not auto-fix truth conflicts during audit.
  • Do not expose sensitive file contents in the final report.
  • Do not treat beautiful structure as proof of correctness.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall