Skip to content

finos/morphir-scala / .claude/skills/kb

Manages the Morphir knowledge base under kb/ — OKF bundles and concept documents. Use when adding content to a bundle, creating a new bundle, checking the knowledge base for conformance or provenance drift, building or querying its SQLite index, managing intent through its lifecycle, recording or reading architectural decisions as Decision Records, or navigating, searching and listing its bundles, concepts and links.

FileKindSize
.gitignoreother180 B
.mill-versionother19 B
KbCheck.scalaother19.7 KB
KbDecision.scalaother12.6 KB
KbIndex.scalaother21.9 KB
KbIntent.scalaother14.0 KB
KbIntentEdit.scalaother10.9 KB
KbModel.scalaother11.7 KB
KbRefresh.scalaother8.9 KB
KbRender.scalaother9.0 KB
KbScaffold.scalaother9.5 KB
KbStore.scalaother12.0 KB
KbSync.scalaother46.9 KB
KbTests.scalaother71.5 KB
SKILL.mdskill md10.8 KB
kbexecutablescript459 B
kb.batscript485 B
kb.scalaother43.3 KB
millexecutablescript6.5 KB
mill.batscript9.7 KB
references/authoring.mdmarkdown4.7 KB
references/checks.mdmarkdown12.0 KB
references/commands.mdmarkdown12.8 KB
references/decisions.mdmarkdown7.4 KB
references/divergence.mdmarkdown3.4 KB
references/index-db.mdmarkdown5.2 KB
references/sync.mdmarkdown16.7 KB
test-resources/META-INF/services/kyo.test.Testother503 B

references/sync.md

View on GitHub
# Mirroring an Upstream Repository

A bundle may declare a `sync.yaml` naming an upstream repository and the paths it mirrors. `kb sync pull` copies
those paths into the bundle, `kb sync push` projects an edited file back to the exact bytes upstream would expect,
and `sync.lock.yaml` records what came from where.

The point is to work on somebody else's material *in place* rather than paraphrasing it. A bundle that only cites
upstream can go stale without anyone noticing; a bundle that carries upstream's own bytes can be diffed, edited and
sent back. The cost is that the copy must stay byte-honest, which is what most of this design is about.

Everything here is offline. `pull`, `push` and `diff` read a local checkout under `.refs/`; nothing reaches the
network, and nothing branches, commits or opens a pull request for you.

## Two markers, doing two jobs

| Marker | Set by | Means |
| ------ | ------ | ----- |
| `sync.yaml` at the bundle root | You, by hand | Files under the manifest's `root:` are **vendored** — mirrored, not authored here. Loading uses this, so checks relax for those files whether or not you ever run `kb sync` |
| `sync: true` in the bundle-root `index.md` | You, by hand | This is the bundle `kb sync` operates on when `--bundle` is not given. The first bundle whose index carries a `sync` key wins |

They are separate because they answer different questions. A bundle with a manifest and no marker still loads
correctly — its mirrored files are still assets and vendored concepts — it just is not the default target of the
commands.

## The manifest

`sync.yaml` is hand-written, reviewed, and the only place upstream coordinates live.

| Key | Default | Means |
| --- | ------- | ----- |
| `upstream.repo` | *required* | `org/repo`, e.g. `finos/morphir` |
| `upstream.ref` | `main` | The branch the mirror tracks. Recorded for the reader — the tooling reads whatever the reference checkout has checked out and pins the resulting commit in the lockfile |
| `upstream.refs_path` | the value of `repo` | Where the checkout sits under `.refs/`, when it is not `<org>/<repo>` |
| `root` | `sources` | The subtree inside the bundle that holds the mirror |
| `mappings` | *required, non-empty* | Globs selecting upstream paths. An entry is either a bare glob or `{ from: <glob>, exclude: [<glob>…] }` |
| `exclude` | none | Globs subtracted from every mapping |
| `type_map` | none | Ordered glob → OKF `type`. **First match wins**, so narrow globs come first. Unmatched paths get `Source Document` |

Globs are matched against `/`-separated upstream-relative paths and support `*` (stops at a separator), `?` and
`**`. `**/` also matches zero directories, so `docs/**/x.md` finds `docs/x.md`.

### `type_map` may not name a register-owned type

A register that discovers its records by `type:` claims that value across the whole knowledge base — `kb decision`
collects every concept whose `type` is `Decision Record` wherever it sits, by design. Injecting one into a mirrored
document conscripts upstream's file into a register whose schema it was never written against: an ADR imported from
`finos/morphir` keeps its id in an `ADR-0001-…` filename and its status in the body, where this register expects a
`NNNN-` prefix and frontmatter, so it fails four checks that nobody on this side can fix.

So a mirrored document is typed by **what it is**, not by which of our registers would like it: `Decision Source`,
not `Decision Record`. `kb sync` refuses a manifest that names an owned type — every command, not just `pull` — and
`kb check` reports the same refusal as `sync-manifest-invalid` so it is caught without a reference checkout. The set
lives in `KbRegisters.ownedTypes`; a register that starts discovering by `type:` adds itself there.

```yaml
# What this bundle mirrors from finos/morphir, and where it lands.
upstream:
  repo: finos/morphir
  ref: main
  refs_path: finos/morphir

root: sources

mappings:
  - "docs/spec/draft/**"
  - "docs/spec/ir/**"
  # The schemas themselves. YAML only: the sibling .json files are generated from these upstream, so mirroring
  # them would mean maintaining two copies of one fact.
  - "website/static/schemas/morphir-*.yaml"
  - { from: "tests/bdd/fixtures/ir/v4/**", exclude: ["**/*.tmp"] }

exclude:
  - "**/node_modules/**"

# First match wins, so the narrower globs come first.
type_map:
  "docs/design/**": Design Source
  "docs/adr/**": Decision Record
  "docs/spec/**": Specification Source
```

## The lockfile

`sync.lock.yaml` is generated by `kb sync pull` and never edited by hand. It records the commit the import came
from and, per file, the SHA-256 of its **upstream** form.

```yaml
# Generated by `kb sync pull`. Do not edit by hand.
base_commit: 4d5e5c06a7cf269c5f86b050a16a6f82bb5c29bc
imported_at: 2026-08-02
files:
  - { path: docs/spec/ir/types.md, kind: concept, upstream_sha256: 9f2b… }
  - { path: website/static/schemas/morphir-ir-v4.yaml, kind: asset, upstream_sha256: 1c7a… }
```

Entries are written in path order, so a pull that changes nothing produces no diff. `path` and `upstream_sha256`
are required on read; `kind` defaults to `concept`.

One hash, not two, and it is the hash of the upstream form. A local file is compared by projecting it first, so
there is no second digest to keep honest and no way for the two to disagree.

## Concepts and assets

| Kind | What lands | Why |
| ---- | ---------- | --- |
| **Concept** — `*.md` | Upstream's bytes plus a fenced block of knowledge-base frontmatter | OKF needs a `type`; the rest of the frontmatter is upstream's |
| **Asset** — everything else | Upstream's bytes, unchanged | It has no frontmatter to hold a `type`, so treating it as a concept would mean every one of them failing the checks that make concepts useful |

`.mdx` is deliberately an asset. It carries JSX, which commonmark has no business parsing, and widening the markdown
predicate to include it would reclassify files in every other bundle at the same time.

Assets are tracked in the lockfile and synced like anything else — they are simply never read as documents. They
appear in `Bundle.assets` rather than `Bundle.concepts`, and in the generated index under an `Assets (n)` heading.

## The fenced block

A mirrored markdown file gains exactly one region the knowledge base owns:

```markdown
---
title: Types
sidebar_position: 2
# kb:begin — added by the knowledge base; removed on export
type: Specification Source
description: "Upstream source document finos/morphir:docs/spec/ir/types.md."
kb_upstream: docs/spec/ir/types.md
# kb:end
---

# Types
```

The injected keys are `type` always, `kb_upstream` always, and `title` and `description` **only when upstream did
not supply them** — a fallback title derived from the filename, and a description naming the repo and path. Nothing
outside the fence is touched.

When upstream has no frontmatter at all, the whole block is ours and the opening fence is marked:

```markdown
---
# kb:begin block — added by the knowledge base; removed on export
type: Source Document
title: Overview
description: "Upstream source document acme/spec:overview.md."
kb_upstream: overview.md
# kb:end
---

# Overview
```

The `block` flag exists because a document whose upstream frontmatter is an *empty* `---` / `---` pair is otherwise
indistinguishable from one that had none, and export would delete a block upstream actually has. The whole block is
removed only when we created it *and* nobody has since added a key of their own alongside it.

### The block is manifest-derived, and self-correcting

Everything inside the fence is *generated*: `type` from `type_map`, `kb_upstream` from the path, `title` and
`description` from the fallbacks. Edit the manifest and the next `kb sync pull` rewrites the block in place — it does
not wait for upstream to touch the file, and it does not need the file to have drifted.

This is not free, and it is worth knowing why it exists. `status` compares *projected* forms, so the injected block is
invisible to it by construction — which is right for detecting upstream drift, and is exactly why nothing used to
notice when our own injection went stale. A clean file was passed over on every pull, so a `type_map` edit had no
effect on anything already imported and the manifest was effectively write-once. Instead of a second hash in the
lockfile, `pull` compares each file against what the manifest now implies and rewrites the ones that disagree:

- The comparison uses the local file alone, so `kb sync status` and `kb check` report staleness with **no reference
  checkout** — a manifest edit that was never applied shows up offline.
- `status` marks such a file `[injection stale]` whatever its state, and `--strict` fails on it. It is not a
  `SyncState`: a file can have drifted upstream *and* carry a stale block, and collapsing the two would lose one.
- `pull` reports the rewrite as its own verb, `re-injected`, and `--dry-run` lists it without writing. A bulk
  re-injection across a whole mirror should not read as an import from upstream.
- Re-injection changes only the fenced region, so the upstream form — and the hash beside it in the lockfile — is
  unchanged. That is why it is safe to do to a `local-only` file: the edit you mean to export is outside the fence.
- **Keys you add inside the fence by hand survive.** `type`, `title`, `description` and `kb_upstream` are ours and are
  recomputed; anything else in there is left alone. The four are fixed rather than taken from the file, because the
  day upstream supplies a `title` of its own, ours has to go or the frontmatter carries the key twice and stops
  parsing.
- A damaged fence is refused, not guessed at. That file is `unreadable` and has its own finding.

### The round-trip invariant

```
project(inject(bytes)) == bytes
```

Byte for byte, including line endings. **This is what makes export safe**: if it slips, the knowledge base silently
rewrites somebody else's repository. `KbSyncSpec` pins it against a deliberately hostile corpus — CRLF throughout,
nested YAML blocks, fractional `sidebar_position`, a `---` in the body, an empty frontmatter block, no trailing
newline, an unterminated fence, no frontmatter at all.

Two consequences follow, and both are load-bearing:

- **Nothing re-serializes YAML.** Upstream frontmatter is moved around as lines, never parsed and rewritten, so a
  fractional number or a nested `tracking:` block cannot be reformatted by accident.
- **Line endings are preserved.** `KbSync.split` deliberately does not normalize CRLF the way `KbStore.splitFrontmatter`
  does, because normalizing is right for parsing and fatal for round-tripping.

A damaged fence — an unmatched `# kb:begin`, an unmatched `# kb:end`, or a close before an open — is refused rather
than guessed at. That file is `unreadable`, and `kb check` reports it as `sync-projection-broken`.

## States

Every run derives state fresh from three inputs: the local file's upstream form, the lockfile's hash, and the
upstream file's hash. Nothing is stored.

| State | Means | `pull` | `push` |
| ----- | ----- | ------ | ------ |
| `clean` | Local upstream form matches the baseline, and upstream has not moved | nothing | nothing |
| `local-only` | Edited here; upstream still at the baseline | leaves it — this is the edit you mean to export | **writes it**, in upstream's form |
| `upstream-only` | Upstream moved; no local edits | takes it — nothing local is lost | nothing |
| `diverged` | Both sides moved since the import | refuses, unless `--theirs` | writes it — see the note below |
| `missing-local` | In the lockfile, absent from the mirror | re-imports it from upstream | nothing |
| `deleted-upstream` | In the lockfile, gone upstream, unmodified here | reports it and keeps it; `--prune` deletes it and drops the entry | nothing |
| `deleted-upstream-edited` | Gone upstream, but edited here since the import | holds it back — neither `--prune` nor `--theirs` will touch it | holds it back; restore it upstream, or drop the edit |
| `untracked` | Upstream has it and a mapping selects it, but the lockfile does not list it | imports it | nothing |
| `unreadable` | The kb fence is damaged, so the file cannot be projected | refuses, unless `--theirs` overwrites it from upstream | skipped |

`push` derives states **without consulting upstream** — it walks the lockfile and compares the mirror against it. So
the only states it can see are `clean`, `local-only`, `missing-local` and `unreadable`; a file that also moved
upstream presents to it as `local-only` and is exported. Run `kb sync status` first if that matters. `push` also
carries a guard for a projection that fails at write time: that path is reported as `refused` and the command exits
non-zero.

`status` considers the union of the lockfile's paths and the upstream paths the manifest selects. A file you add
into the mirror by hand is in neither, so sync will not see it — import it through `pull`, or it will not be
tracked.

A manifest path that would escape the mirror (leading `/`, or a `..` segment) is refused outright, the same guard
`add-concept` carries.

## Commands

Full flag tables live in [commands.md](commands.md); this is what each one is for.

**`kb sync status`** — what has moved, on both sides. It degrades gracefully: with no reference checkout, or with
`--no-upstream`, it compares the mirror against the lockfile alone and reports only the states that comparison can
decide. `--verbose` lists clean files too; `--strict` exits non-zero when anything is `diverged`, `unreadable`, or
carrying a stale injected block — that last one is a manifest edit that was never applied, which is a manifest true
only on paper.

**`kb sync pull`** — import. Requires the reference checkout. It writes the mirrored files, re-injects any block that
no longer matches the manifest, rewrites `sync.lock.yaml` with the checkout's current HEAD as `base_commit`, then
regenerates the bundle index below the `<!-- kb:sources -->` marker. `--dry-run` reports and writes nothing;
`--theirs` takes upstream's version of files that changed on both sides; `--prune` deletes what upstream has removed.

The index is generated because forty hand-written bullets rot on the first pull. Everything above the marker is
yours; everything below it is rewritten. Bullet text is the concept's own `description`, verbatim — that equality is
what `index-description-drift` enforces, so generating it from anything else would break the check on import.

**`kb sync push`** — export. Projects each exportable file to its upstream form and writes it into a checkout,
defaulting to the reference checkout and overridable with `--to`. It writes files and stops: branching, validating
and opening a pull request are separate, explicit steps.

**`kb sync diff <path>`** — `git diff` between upstream's copy and the **upstream form** of ours, which is the thing
an export would actually send. Take the path from `sync status`; it is relative to the mirror root, e.g.
`docs/spec/draft/types.md`.

## Checks

`kb check` folds sync findings in alongside the rest, for every bundle that has a `sync.yaml`. Drift is a prompt,
not a failure — only `sync-projection-broken`, `sync-lock-drift` and `sync-manifest-invalid` are errors: the first
two are the states in which an export would send the wrong bytes, and the third is a manifest no command will accept.
A stale injected block is `sync-injection-stale`, a warning, because `kb sync pull` fixes it mechanically.

Mirrored documents also get relaxed structural checks: their frontmatter is upstream's, so `title`, `description`,
`status` and unknown-key findings are suppressed, and a broken link becomes `link-broken-upstream` (a warning)
rather than `link-broken`. `type` is still required — `pull` injects it, so its absence means the injection failed.

→ [checks.md](checks.md) for the catalogue and the fixes.

## Relationship to `sources:`

Provenance through the `sources:` frontmatter family is the older, lighter mechanism, and it is not superseded.

| | `sources:` | `sync.yaml` |
| --- | --- | --- |
| What is in the repo | A commit-pinned URL | Upstream's bytes |
| Granularity | Per concept | Per mirrored file |
| Detects drift by | Comparing the pin to `.refs/<org>/<repo>` HEAD | Comparing hashes of the upstream form |
| Writes upstream | Never | `kb sync push` |
| Cost | A line of frontmatter | A subtree, a lockfile, and a review obligation |

Reach for `sources:` when a concept **interprets** upstream — which is most of the time. A concept is worth writing
precisely because it says something the source does not. Reach for a mirror when you need upstream's own text in the
repo: to quote it exactly, to diff two versions of it, or to edit it and send the edit back.

The two compose. A mirrored bundle holds the material; the sibling bundles that cite it hold the reading of it, and
their `sources:` entries can pin the same commit the lockfile records.
deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall