Skip to content

majiayu000/spellbook / skills/codebase-audit

codebase-audit

Verified

全面代码库审计 — 自适应并行深度分析(前后端契约、数据完整性、异常处理/安全、架构/技术债、配置/缓存),结构化 findings + 对抗验证 + 基线对比,输出按严重程度排序的统一报告和修复路线图。支持 quick 快速体检模式。Use when user asks to audit, analyze, or review an entire codebase for design issues, find hidden bugs, check architecture health, or asks '全面审查', '代码库审计', '分析设计问题', 'audit codebase', 'health check', '有哪些问题', '快速体检'. Also trigger when user asks to find silent degradation, data flow breakpoints, type mismatches between frontend and backend, or wants to understand technical debt across a project.

evals/expected-findings.json

View on GitHub
{
  "fixture": "evals/fixture",
  "scoring": "Match by file + category (line numbers may drift). Pass threshold: >= 8 of the 10 must_detect findings reported. Bonus findings raise confidence but are not required.",
  "planted_findings": [
    {
      "id": 1,
      "must_detect": true,
      "category": "registry-coverage",
      "file": "app/registry.py",
      "description": "SECTION_BUILDERS has 5 keys but SECTION_META has 3 — render_section returns None for faq/testimonials, which build_page silently drops"
    },
    {
      "id": 2,
      "must_detect": true,
      "category": "silent-degradation",
      "file": "app/handlers.py",
      "description": "sync_metrics: except Exception: pass — failed metric events silently discarded"
    },
    {
      "id": 3,
      "must_detect": true,
      "category": "silent-degradation",
      "file": "app/handlers.py",
      "description": "load_section: except Exception -> logger.warning + return DEFAULT_SECTION — user-visible fallback content at warning level, callers treat it as success"
    },
    {
      "id": 4,
      "must_detect": true,
      "category": "cache-integrity",
      "file": "app/pipeline.py",
      "description": "cache_key only includes section_name + payload — no code/schema/prompt version dimension, stale entries survive logic changes"
    },
    {
      "id": 5,
      "must_detect": true,
      "category": "security",
      "file": "app/config.py",
      "description": "API_KEY hardcoded as a string literal"
    },
    {
      "id": 6,
      "must_detect": true,
      "category": "serialization",
      "file": "app/pipeline.py",
      "description": "SectionPayload extra=\"ignore\" silently drops unknown fields, and serialize() uses model_dump(exclude_none=True) on the cache write path"
    },
    {
      "id": 7,
      "must_detect": true,
      "category": "declaration-execution",
      "file": "app/config.py",
      "description": "Settings class declared but never instantiated anywhere — main.py loads yaml directly and uses os.getenv"
    },
    {
      "id": 8,
      "must_detect": true,
      "category": "config-completeness",
      "file": "app/main.py",
      "description": "_cfg.get(\"retry_limit\", 3) — retry_limit is used in code but absent from config.yaml, silently defaults"
    },
    {
      "id": 9,
      "must_detect": true,
      "category": "test-quality",
      "file": "tests/test_pipeline.py",
      "description": "test_serialize_roundtrip has vacuous assertions (is not None + assert True); the only meaningful test is @pytest.mark.skip'd",
      "note": "requires tests dimension (Agent 6) enabled; when scoring a run without it, replace with bonus finding 11 in the must set"
    },
    {
      "id": 10,
      "must_detect": true,
      "category": "concurrency",
      "file": "app/tasks.py",
      "description": "run_job is async def but calls time.sleep (blocks the event loop); _job_counter incremented without a lock",
      "note": "requires concurrency dimension (Agent 7) enabled; when scoring a run without it, replace with bonus finding 12 in the must set"
    },
    {
      "id": 11,
      "must_detect": false,
      "category": "declaration-execution",
      "file": "app/storage.py",
      "description": "JobStore.persist/restore never called by any startup or shutdown path — jobs lost on restart"
    },
    {
      "id": 12,
      "must_detect": false,
      "category": "resource-cleanup",
      "file": "app/storage.py",
      "description": "export_report: temp file not removed if upload() raises (no try/finally); also uses insecure tempfile.mktemp"
    }
  ]
}
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall