Skip to content

opensiro/vsm-harness-skills / skills/build-harness-cognee-first

build-harness-cognee-first

Verified

Build a new agent harness through verifiable Cognee-first architectural discovery, source inspection, engineering decisions, and traceable implementation; runtime benchmarking is optional.

Cognee-first harness construction

Skill version: 0.1.0

Status: experimental draft v0.1.0; reusable build-time workflow, not a released VSM assessment specification.

Purpose and scope

Use this skill for a new harness construction experiment, not for producing or changing a canonical VSM assessment. Choose and implement an architecture from actual build-time research: define a task, retrieve a pinned assessment corpus through Cognee, inspect upstream code, compare alternative mechanisms, record the architecture decision, and produce an inspectable new harness. The experiment concerns the harness development cycle, not game-playing or model performance. Cognee must not become an implicit runtime dependency.

This is an experimental build-time skill in opensiro/vsm-harness-skills. It is independently versioned and not a release of the canonical VSM assessment Methodology. It does not amend assess-vsm-harness, the VSM Profile, or the Index admission rules. Its historical Flash API case illustrates process and failure modes, not a recommended architectural outcome.

Stage 0 — task and source preflight

  1. Read the current public Opensiro bootstrap and follow its owning-repository routing.
  2. Declare the harness construction task, acceptance criteria for the built artifact, allowed source/tools, available build-time resources, cost/time limits, and a minimal starting scaffold. Model execution and benchmark design are optional. Do not preselect an adapted mechanism or upstream winners.
  3. Pin the Index Git commit, connector commit, assessment/Profile dependency versions, and available public sources. Verify canonical inclusion against the pinned Index. Record all pins and connector fixes. Preserve historical references; do not silently upgrade frozen sources.
  4. Plan dataset selection by a stated, reproducible rule before retrieval; disclose exclusions and how the selection may miss relevant systems. Prefer the entire included corpus when feasible. Sampling is allowed only with recorded compute/storage limits and a deterministic selection policy. The historical 128-record sample is not a default or target.
  5. Freeze a task-specific run contract: construction outcome criteria, starting scaffold, OS/runtime/dependencies, optional model/API configuration, resource budgets, Cognee coverage, planned source/static validation, permissions, and safe construction boundary. Never record credentials in receipts.
  6. Publish C1 — preflight checkpoint: source pins, admission/sample policy, task/baseline contract and preregistered query plan at an immutable Git commit before the first Cognee import, dataset-index build or corpus-inspecting discovery action, not merely before the first native search query. Record the exact commit SHA. A pre-C1 source-inventory inspection strictly necessary to choose and pin the dataset must be logged as preflight, with no shortlist or architectural choice. If no shared remote can publish, preserve local Git ancestry and label the weaker evidence explicitly; do not claim independently verifiable public precommit.

Gate 1 — execute real Cognee discovery

  1. Before construction, prepare a fresh, isolated Cognee dataset, respecting the connector's fresh-only and failed-attempt restrictions.
  2. Use the pinned OpenSiro Cognee connector to select, resolve and import the included assessment records into a real Cognee dataset. Record the connector repository/commit, exact package/entry point invoked, compatibility patches with hashes, source/document IDs, accepted/rejected counts and original connector/import receipts. A custom direct-SDK importer or pasted assessment text does not satisfy this connector-first gate; label such a run a separate experiment or a failed/partial connector gate. Never claim connector usage from a pinned SHA alone.
  3. Independently audit the resulting dataset's source identity, provenance, graph facts, and exact literal autonomy grades. A failed audit blocks Cognee-dependent claims.
  4. Use only the previously frozen query plan from C1 for the first discovery pass; any later query expansion must be separately identified and timestamped. Run task-grounded queries and at least one useful negative/control query using the actual native Cognee search API over the freshly connector-imported dataset, recording query IDs, search mode, indexed fields/chunks, raw returned hits/ranks, irrelevant results, errors and empty hits. Preserve separately the connector ingestion receipt and native Cognee query receipt; one is not evidence for the other. Do not substitute grep, a cached answer, a mock graph, direct-SDK-only ingestion, or a retrospective replay for a passing connector-first gate.
  5. Explicitly distinguish:
    • Ingested source: which full assessment documents are stored.
    • Indexed retrieval: exactly which sections/text/fields the search can return.
    • Later source inspection: which complete assessments and upstream files are actually read. The first example ingested full Markdown but indexed only six S1–S5 function sections for lexical search. Its TLDR.md was not the search corpus. Never present section-only lexical retrieval as full-document search.
  6. If a broader architectural search needs full-body indexing, implement and validate it as a new experiment/dataset. Do not claim it happened in the historical case. If native ingestion, audit, or search is blocked, stop Gate 1 and report partial work without inventing results.

Gate 1 pass / C2 — discovery checkpoint: a C1 Git anchor predating first connector import, original connector invocation/import receipts, independent audit PASS, the frozen C1 query plan, actual native Cognee query results including negative and irrelevant hits, and explicit searchable-scope documentation have been saved and published with a Git commit SHA before candidate comparison and mechanism selection. A document being ingested does not imply its entire body was indexed. Preserve the failed original attempt if repair/retry is needed, and require a fresh isolated dataset for another attempt.

Gate 2 — source-backed shortlist and design

  1. Only after Gate 1, derive a shortlist from its actual hits. Consider additional public research/GitHub evidence explicitly as outside-Index sources, not fabricated indexed hits or VSM grades.
  2. Read the full canonical assessment for each shortlisted system; then inspect the relevant upstream implementation at its actual pinned reviewed SHA. Record file/line links and evidence for the claimed mechanism. Relevance in one chunk is not implementation proof.
  3. Compare implementation fit, complexity, architecture tradeoffs, licensing, failure modes, and source-evidence strength. Consider minimal/no-adaptation construction, alternatives, and rejected choices. Neither S4 presence nor A/C/P values are performance rankings; feedback inside one task is not automatically S4.
  4. Publish a candidate comparison and alternatives at a pinned Git commit before choosing a mechanism. Include which Cognee result produced each candidate, the source files actually inspected, and an honest coverage gap for potentially missed candidates. Distinguish candidates found by Cognee from candidates first found through other public research; never relabel external discovery as a Cognee hit.
  5. Record the architecture decision at a subsequent distinct Git commit referring to the comparison SHA, with selected/rejected options, source support, and the evidence-backed Cognee discovery contribution (DIRECT / CORROBORATIVE / NONE / INDETERMINATE). Select the smallest justified adaptation, or explicitly select none. Do not cherry-pick code or retroactively edit the comparison to imply prior selection.

Gate 2 pass / C3 — decision checkpoint: traceable retrieved hits → complete assessments and pinned GitHub inspection → published comparison → subsequent decision commit. The comparison commit must be an ancestor of the decision commit. The decision commit must precede implementation of the selected mechanism; check Git ancestry and record these SHAs. If the decision is no-adaptation, record the null outcome without fabricating an adapted-code commit. Existing case-study conclusions are not reusable as fresh findings.

Stage 3 — construct the harness

  • After C3 implement the chosen bounded mechanisms and record exactly which source-backed decisions shaped the new files. A minimal fixture/scaffold may precede Cognee discovery only when it does not preselect a preferred adaptation. NO_ADAPTATION still requires a produced, inspectable minimal harness satisfying the predeclared construction criteria; it is a legitimate design decision, not permission to omit the artifact.
  • Publish inspectable harness source, dependencies and entry points; link each adapted mechanism to its source, decision ID, and local implementation. Make licensing and excluded features explicit.
  • Use source inspection, AST/static checks, dependency checks or ordinary unit tests only if permitted. Keep the meaning of each verification method separate from runtime functionality claims.
  • Do not require running the harness, a gameplay episode, a model API call, a baseline/adapted performance benchmark, a win rate, or model-quality improvement to finish the development-cycle demonstration.
  • If runtime execution is not authorized, preserve NOT_RUN and do not trigger it indirectly through CI. Never treat an unexecuted implementation as proven functionally working.

C4 — constructed artifact and provenance checkpoint

Publish the resulting harness source, even for a NO_ADAPTATION decision, with dependencies/entry points, a source-to-code mapping (or explicit no-added-mechanism explanation), source/static validation if actually performed, remaining risks, and artifact hashes at a commit after C3. Verify ancestry. If no harness artifact was constructed, report build_status=PARTIAL or FAIL as warranted and process_status not PASS; preserve the research as a bounded partial result, not a completed C4.

The primary C4 evidence is Cognee output → inspected upstream source → alternatives → recorded decision → produced harness code. Runtime and model evaluations are optional follow-ups, not a mandatory C4 gate. Preserve historical receipts; never retrofit chronology. Git commits prove public artifact order rather than private intent or performance.

Mandatory observable Decision Trace

Preserve at least: task definition; pinned source and coverage; precommitted dataset/query plan; import/audit; raw native query receipts; shortlisted and rejected candidates; full-source inspection; explicit architectural decision; produced implementation; verification if actually performed; and subsequent corrections.

Use an append-only event log with timestamps, source references, decision IDs, rejected options, related commits and artifact paths. Record what was actually observed; do not invent hidden reasoning or re-time earlier actions. See Decision Trace contract.

Completion states and evidence strength

The primary measured object is the development process, not the resulting harness's win rate or model performance. Report independently:

  • process_status: PASS only when C1→C2→C3→C4 are evidenced chronologically; otherwise PARTIAL or FAIL with missing gates.
  • build_status: PASS if the constructed harness source and its design-source mapping (or supported NO_ADAPTATION explanation) meet the predeclared acceptance criteria and are inspectable; PARTIAL / FAIL if missing or incomplete. A source-complete build is not automatically runtime-verified. No built harness means no full C4 PASS.
  • verification_level: SOURCE_ONLY, STATIC_CHECKED, UNIT_TESTED, RUNTIME_SMOKE, or NOT_RUN; report only levels actually observed.
  • evaluation_level: NOT_RUN is acceptable. Optional DETERMINISTIC_ONLY / SCRIPTED_PROTOCOL / REAL_MODEL_RUN may be documented separately.
  • model_quality_claim: NOT_EVALUATED unless a separately authorized and appropriately matched real-model evaluation exists.
  • discovery_contribution: DIRECT (Cognee first surfaced the selected mechanism and the decision cites its actual retrieval hit), CORROBORATIVE (Cognee independently confirmed a mechanism already found elsewhere), NONE (no influence on the design choice, including a justified no-adaptation outcome), or INDETERMINATE (evidence cannot establish the source of influence). Cite query/hit IDs, alternative discovery channel and decision ID. Only DIRECT supports a claim of Cognee-driven mechanism discovery; CORROBORATIVE supports a narrower corroboration claim. Even when process_status=PASS, NONE/INDETERMINATE must not be presented as Cognee-influenced architecture.

A successful Cognee-first construction trace does not depend on running the constructed harness. Strong gameplay results also cannot substitute for missing Cognee-first evidence.

Deliverables and completion

  • Declared construction task, constraints, source pins, dependencies, and an inspectable produced harness.
  • Cognee connector import, independent audit and raw native search receipts with exact indexed-vs-ingested scope.
  • Full assessments and pinned upstream code inspection, compared alternatives and a decision made before adapted code.
  • A trace linking source → connector import → native Cognee retrieval → comparison → decision → constructed harness; frozen C1/C2/C3/C4 Git anchors and an explicit discovery_contribution classification.
  • Optional source/static/runtime checks labeled accurately; agent/model evaluation is not required.

Full development-process success requires both discovery/source-selection gates and a produced harness artifact, not a game or coding task success rate. A justified no-adaptation choice can achieve PASS only if a complete minimal harness is delivered. Stopping after source review without constructing the harness is PARTIAL/FAIL, never a completed build. Any unexecuted harness is NOT_RUN for runtime correctness; an incomplete Cognee gate remains a process failure even if the source code appears useful.

reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall