Skip to content

finos/morphir-scala / .claude/skills/kb

Manages the Morphir knowledge base under kb/ — OKF bundles and concept documents. Use when adding content to a bundle, creating a new bundle, checking the knowledge base for conformance or provenance drift, building or querying its SQLite index, managing intent through its lifecycle, recording or reading architectural decisions as Decision Records, or navigating, searching and listing its bundles, concepts and links.

FileKindSize
.gitignoreother180 B
.mill-versionother19 B
KbCheck.scalaother19.7 KB
KbDecision.scalaother12.6 KB
KbIndex.scalaother21.9 KB
KbIntent.scalaother14.0 KB
KbIntentEdit.scalaother10.9 KB
KbModel.scalaother11.7 KB
KbRefresh.scalaother8.9 KB
KbRender.scalaother9.0 KB
KbScaffold.scalaother9.5 KB
KbStore.scalaother12.0 KB
KbSync.scalaother46.9 KB
KbTests.scalaother71.5 KB
SKILL.mdskill md10.8 KB
kbexecutablescript459 B
kb.batscript485 B
kb.scalaother43.3 KB
millexecutablescript6.5 KB
mill.batscript9.7 KB
references/authoring.mdmarkdown4.7 KB
references/checks.mdmarkdown12.0 KB
references/commands.mdmarkdown12.8 KB
references/decisions.mdmarkdown7.4 KB
references/divergence.mdmarkdown3.4 KB
references/index-db.mdmarkdown5.2 KB
references/sync.mdmarkdown16.7 KB
test-resources/META-INF/services/kyo.test.Testother503 B

KbCheck.scala

View on GitHub
//| scalaVersion: 3.9.0
//| moduleDeps: [KbStore.scala, KbDecision.scala]
//| mvnDeps:
//| - io.getkyo::kyo-core:1.0.0-RC7

/** Conformance and drift checks over a knowledge base.
  *
  * Two families:
  *   - structural — does the KB obey OKF and the conventions in kb/AGENTS.md?
  *   - provenance — do the commit-pinned `sources` still line up with the reference checkouts under `.refs/`?
  *
  * Everything is offline. Provenance runs `git` against local checkouts; it never reaches the network.
  */

import kyo.*
import java.time.LocalDate

// `Severity` and `Finding` live in KbModel.scala — they are pure data, and putting them here would force every
// module that merely produces findings (KbIntent, KbDecision) to depend on the check runner.

object KbCheck:

  private val GitHubUrl = raw"https://github\.com/([^/]+)/([^/]+)/(?:blob|tree)/([0-9a-f]{7,40})/(.*)".r

  /** Runs every check. `today` is injected so results are reproducible. */
  def run(
      kb: Kb,
      refsRoot: Option[Path],
      today: LocalDate,
      allowDangling: Boolean = false
  ): Seq[Finding] < (Async & Abort[Throwable]) =
    for
      structural <- Kyo.foreach(kb.bundles)(bundleFindings(kb, _, today, allowDangling))
      provenance <- refsRoot match
        case None => (Chunk.empty[Chunk[Finding]]: Chunk[Chunk[Finding]] < (Async & Abort[Throwable]))
        case Some(r) => Kyo.foreach(kb.concepts)((_, d) => sourceFindings(kb, d, r))
    yield
      val strays = kb.strays.map { p =>
        Finding(
          Severity.Error,
          "stray-markdown",
          kb.rel(p),
          None,
          "markdown file under bundles/ that belongs to no bundle",
          Some("a bundle root is a directory whose index.md carries okf_version; grouping directories use README.md")
        )
      }
      (structural.flatten.toSeq ++ strays ++ KbDecision.findings(kb) ++ provenance.flatten.toSeq)
        .sortBy(f => (f.severity.ordinal, f.path, f.line.getOrElse(0)))

  private def bundleFindings(kb: Kb, b: Bundle, today: LocalDate, allowDangling: Boolean): Chunk[Finding] < (Sync & Abort[Throwable]) =
    Kyo.foreach(b.allDocs)(docFindings(kb, b, _, today, allowDangling)).map { perDoc =>
      Chunk.from(perDoc.flatten) ++
        Chunk.from(indexCoverage(kb, b) ++ indexDescriptions(kb, b) ++ duplicateTitles(kb, b) ++ readmeInBundle(kb, b))
    }

  private def docFindings(kb: Kb, b: Bundle, d: Doc, today: LocalDate, allowDangling: Boolean): Chunk[Finding] < (Sync & Abort[Throwable]) =
    linkFindings(kb, b, d, allowDangling).map(links => Chunk.from(pureDocFindings(kb, d, today)) ++ links)

  /** Everything about a document that can be decided without touching the filesystem. */
  def pureDocFindings(kb: Kb, d: Doc, today: LocalDate): Seq[Finding] =
    val where = kb.rel(d.file)

    val fmErrors = d.frontmatterError.toSeq.map { msg =>
      Finding(Severity.Error, "frontmatter-invalid", where, Some(1), s"YAML did not parse: $msg")
    }

    val kindErrors = d.kind match
      case DocKind.SubIndex if d.hasFrontmatterBlock =>
        Seq(Finding(
          Severity.Error,
          "subindex-has-frontmatter",
          where,
          Some(1),
          "a non-root index.md must have no frontmatter",
          Some("only the bundle-root index.md carries frontmatter, and there only okf_version and friends")
        ))
      case DocKind.Concept if !d.hasFrontmatterBlock =>
        Seq(Finding(
          Severity.Error,
          "concept-no-frontmatter",
          where,
          Some(1),
          "concept document has no frontmatter block",
          Some("every concept needs at least `type:`")
        ))
      case _ => Nil

    // A mirrored document's frontmatter belongs to upstream. Demanding OKF's `status` vocabulary of it, or reporting
    // every `sidebar_position` as an unknown key, would bury the findings that are actually ours to fix. `type` still
    // has to be there — `kb sync pull` injects it, so its absence means the injection failed.
    val vendoredErrors =
      if !d.vendored || !d.hasFrontmatterBlock then Nil
      else
        Seq(
          Option.when(d.fm.`type`.isEmpty)(
            Finding(
              Severity.Error,
              "concept-missing-type",
              where,
              Some(1),
              "mirrored concept has no `type` — the kb frontmatter block is missing or damaged",
              Some("re-run `kb sync pull` for this bundle")
            )
          )
        ).flatten

    val conceptErrors =
      if !d.isConcept || !d.hasFrontmatterBlock || d.vendored then Nil
      else
        val fm = d.fm
        Seq(
          Option.when(fm.`type`.isEmpty)(
            Finding(Severity.Error, "concept-missing-type", where, Some(1), "concept has no `type` — the one universally required OKF field")
          ),
          Option.when(fm.title.isEmpty)(Finding(Severity.Warn, "concept-missing-title", where, Some(1), "concept has no `title`")),
          Option.when(fm.description.isEmpty)(
            Finding(Severity.Warn, "concept-missing-description", where, Some(1), "concept has no `description` — indexes and search snippets read it")
          ),
          fm.status.filterNot(Frontmatter.Statuses.contains).map(s =>
            Finding(Severity.Warn, "status-unknown", where, Some(1), s"status `$s` is not one of ${Frontmatter.Statuses.toSeq.sorted.mkString(", ")}")
          ),
          fm.staleAfter.flatMap(parseDate).filter(_.isBefore(today)).map(dt =>
            Finding(Severity.Warn, "stale-after-passed", where, Some(1), s"stale_after $dt has passed — content is due for review")
          )
        ).flatten ++ unknownKeys(where, d)

    fmErrors ++ kindErrors ++ conceptErrors ++ vendoredErrors ++ escapingLinks(where, d) ++ figureFindings(where, d)

  /** Figures — mermaid fences and standalone images — carry numbered captions: a paragraph directly after the
    * figure starting `**Figure N:**`, numbered 1..N in document order. Prose cites a figure by number in text
    * ("see Figure 2"); no HTML anchors are used, because no anchor form renders reliably across the renderers
    * the kb meets. The bold prefix is the convention; an unbolded `Figure N:` is accepted.
    *
    * Only this flat per-document scheme is checked. Section-aware numbering (`Figure 2.1`) is deliberately out of
    * scope for now; a caption in such a scheme reports as out-of-sequence rather than passing unvalidated.
    */
  private val FigureCaption =
    raw"""\*{0,2}[Ff]igure\s+(\d+)(?:\s*[:.]\s*\*{0,2}|\*{0,2}\s*[:.]).*""".r

  private def figureFindings(where: String, d: Doc): Seq[Finding] =
    if !d.isConcept || d.vendored then Nil
    else
      val lines = d.body.linesIterator.toVector
      val offset = d.frontmatterLines
      val findings = Vector.newBuilder[Finding]
      var expected = 1

      def checkCaption(figureEndIdx: Int, label: String): Unit =
        var j = figureEndIdx + 1
        while j < lines.length && lines(j).trim.isEmpty do j += 1
        lines.lift(j).map(_.trim) match
          case Some(FigureCaption(n)) =>
            if n.toInt != expected then
              findings += Finding(
                Severity.Warn,
                "figure-number-out-of-sequence",
                where,
                Some(j + 1 + offset),
                s"$label caption is numbered Figure $n but this is figure $expected of the document",
                Some("number figures 1..N in document order; section-aware schemes are not yet supported")
              )
          case _ =>
            findings += Finding(
              Severity.Warn,
              "figure-caption-missing",
              where,
              Some(figureEndIdx + 1 + offset),
              s"$label has no numbered caption",
              Some(s"follow the figure with a caption paragraph: `**Figure $expected:** <what to notice>`")
            )
        expected += 1

      var i = 0
      var openFence: Option[String] = None
      while i < lines.length do
        val t = lines(i).trim
        openFence match
          case Some(info) =>
            if t.startsWith("```") then
              if info == "mermaid" then checkCaption(i, "mermaid diagram")
              openFence = None
          case None =>
            if t.startsWith("```") then openFence = Some(t.drop(3).trim.takeWhile(!_.isWhitespace).toLowerCase)
            else if t.startsWith("![") && t.endsWith(")") then checkCaption(i, "image")
        i += 1
      findings.result()

  /** A bundle-relative link starts at the bundle root, so a `..` segment in one can only take it outside the bundle.
    *
    * This needs its own check because such a link usually still *resolves*: `resolveLink` joins the destination onto
    * the bundle root verbatim, and the filesystem then collapses the `..` when the path is tested for existence. So
    * `link-broken` stays silent while the link means something other than it says, and will break as soon as either
    * bundle moves. Cross-bundle references should use an ordinary relative path.
    */
  private def escapingLinks(where: String, d: Doc): Seq[Finding] =
    d.links.filter(_.isBundleRelative).filter { l =>
      val segments = l.dest.takeWhile(_ != '#').stripPrefix("/").split('/')
      segments.foldLeft((0, false)) {
        case ((depth, escaped), "" | ".") => (depth, escaped)
        case ((depth, escaped), "..") => if depth == 0 then (0, true) else (depth - 1, escaped)
        case ((depth, escaped), _) => (depth + 1, escaped)
      }._2
    }.map { l =>
      Finding(
        Severity.Error,
        "link-escapes-bundle",
        where,
        Some(l.line),
        s"bundle-relative link climbs above the bundle root: ${l.dest}",
        Some("bundle-relative paths start at the bundle root; use a normal relative path to reach another bundle")
      )
    }

  private def unknownKeys(where: String, d: Doc): Seq[Finding] =
    d.fm.values.keys.filterNot(Frontmatter.isRecognized).toSeq.sorted.map { k =>
      Finding(Severity.Info, "frontmatter-unknown-key", where, Some(1), s"frontmatter key `$k` is recognized by neither OKF v0.2 nor this tooling")
    }

  private def parseDate(s: String): Option[LocalDate] =
    try Some(LocalDate.parse(s.trim)) catch case _: Exception => None

  /** OKF says *consumers* must tolerate dangling links as not-yet-written knowledge. This is a producer-side
    * linter, where a dangling link is nearly always a typo — so it is an error by default. `allowDangling` restores
    * OKF's lenient stance for a knowledge base that genuinely links forward to unwritten work.
    */
  private def linkFindings(kb: Kb, b: Bundle, d: Doc, allowDangling: Boolean): Chunk[Finding] < (Sync & Abort[Throwable]) =
    Path.runReadOnly {
      // A mirrored document's links are written for the *upstream site*, where `../migration-guide/` and
      // `/schemas/x.yaml` are routes rather than paths on disk. Resolving those needs the site's own configuration,
      // which a domain-neutral tool cannot know — so only relative links that name an actual file are checked, and the
      // rest are left to whatever link checker upstream runs. Reporting routes as rot would bury the real findings.
      val resolveOne: LinkRef => Option[Path] =
        if d.vendored then
          link =>
            val target = link.dest.takeWhile(_ != '#')
            val last = target.split("/").filter(_.nonEmpty).lastOption
            val namesAFile = last.exists(s => s.contains('.') && s != "." && s != "..")
            Option.when(namesAFile && !link.isBundleRelative)(link).flatMap(KbStore.resolveLink(d, _))
        else link => KbStore.resolveLink(d, link)

      val severity =
        if d.vendored || allowDangling then Severity.Warn else Severity.Error
      val rule = if d.vendored then "link-broken-upstream" else "link-broken"

      val resolvable = d.links.flatMap(l => resolveOne(l).map(l -> _))
      Kyo.foreach(Chunk.from(resolvable)) { (link, target) =>
        target.exists.map {
          case true => Chunk.empty[Finding]
          case false =>
            Chunk(Finding(
              severity,
              rule,
              kb.rel(d.file),
              Some(link.line),
              s"link target does not exist: ${link.dest}",
              if d.vendored then Some("upstream's own link; fix it there and export, or leave it")
              else Option.when(link.isBundleRelative)("bundle-relative paths start at the bundle root, not the kb root")
            ))
        }
      }.map(_.flattenChunk)
    }

  /** Every concept should be reachable from an index in its own bundle. */
  private def indexCoverage(kb: Kb, b: Bundle): Seq[Finding] =
    val linked: Set[String] = b.allIndexes.flatMap(_.links.filter(_.isBundleRelative).map(_.dest.takeWhile(_ != '#'))).toSet
    b.concepts.filterNot(c => linked.contains(c.bundlePath)).map { c =>
      Finding(
        Severity.Warn,
        "concept-not-indexed",
        kb.rel(c.file),
        None,
        s"concept is not linked from any index in ${b.label}",
        Some(s"add `* [${c.displayTitle}](${c.bundlePath}) - ${c.fm.description.getOrElse("…")}` to an index")
      )
    }

  /** Index bullets mirror the target's `description`; drift between them is a real inconsistency. */
  private def indexDescriptions(kb: Kb, b: Bundle): Seq[Finding] =
    b.allIndexes.flatMap { idx =>
      idx.body.linesIterator.zipWithIndex.flatMap { (line, i) =>
        KbMarkdown.IndexEntry.findFirstMatchIn(line).toSeq.flatMap { m =>
          val dest = m.group(3)
          val text = Option(m.group(4)).map(_.trim).filter(_.nonEmpty)
          if !dest.startsWith("/") then Nil
          else
            b.conceptAt(dest.takeWhile(_ != '#')).toSeq.flatMap { target =>
              (text, target.fm.description.map(_.trim)) match
                case (Some(t), Some(dsc)) if normalize(t) != normalize(dsc) =>
                  Seq(Finding(
                    Severity.Warn,
                    "index-description-drift",
                    kb.rel(idx.file),
                    Some(i + 1),
                    s"index entry text differs from the concept's description ($dest)",
                    Some(s"concept says: $dsc")
                  ))
                case _ => Nil
            }
        }
      }.toSeq
    }

  private def normalize(s: String): String =
    s.trim.stripSuffix(".").replaceAll("\\s+", " ").toLowerCase

  /** Mirrored documents are excluded: upstream reuses `index.md` and `whats-new.md` across version directories, and
    * that is upstream's naming to answer for, not a defect in this bundle.
    */
  private def duplicateTitles(kb: Kb, b: Bundle): Seq[Finding] =
    b.authoredConcepts.groupBy(_.displayTitle).filter(_._2.size > 1).toSeq.sortBy(_._1).flatMap { (title, docs) =>
      docs.map(d =>
        Finding(Severity.Warn, "duplicate-title", kb.rel(d.file), Some(1), s"title `$title` is used by ${docs.size} concepts in ${b.label}")
      )
    }

  private def readmeInBundle(kb: Kb, b: Bundle): Seq[Finding] =
    b.authoredConcepts.filter(_.name.equalsIgnoreCase("README.md")).map { d =>
      Finding(
        Severity.Error,
        "readme-in-bundle",
        kb.rel(d.file),
        None,
        "README.md inside a bundle is parsed as a concept document",
        Some("put bundle orientation in index.md; README.md belongs to grouping directories only")
      )
    }

  // ---------------------------------------------------------------- provenance

  /** Current HEAD of a local checkout, or None when it is not a git repository. */
  def gitHead(repo: Path): Option[String] < (Async & Abort[Throwable]) =
    Abort.run[Throwable](
      Command("git", "-C", KbPath.render(repo), "rev-parse", "HEAD").textWithExitCode
    ).map {
      // Validate by shape rather than by exit code: `kyo.Process.ExitCode` is opaque, and a successful
      // `git rev-parse HEAD` is exactly one 40-character hex SHA. Anything else means "not a git checkout".
      case Result.Success((out, _)) if out.trim.matches("[0-9a-f]{40}") => Some(out.trim)
      case _ => None
    }

  /** Compares each commit-pinned GitHub source against the matching checkout under `.refs/<org>/<repo>`. */
  private def sourceFindings(kb: Kb, d: Doc, refsRoot: Path): Chunk[Finding] < (Async & Abort[Throwable]) =
    Path.runReadOnly {
      val where = kb.rel(d.file)
      val refsName = refsRoot.parts.lastOption.getOrElse(".refs")
      Kyo.foreach(Chunk.from(d.fm.sources)) { src =>
        src.resource match
          case GitHubUrl(org, repo, sha, path) =>
            val checkout = refsRoot / org / repo
            checkout.exists.map {
              case false =>
                Chunk(Finding(
                  Severity.Info,
                  "source-ref-missing",
                  where,
                  None,
                  s"no reference checkout for $org/$repo under $refsName/",
                  Some("add one so provenance can be verified: /squire reference repo add")
                ))
              case true =>
                val cleanPath = path.takeWhile(_ != '#')
                for
                  head <- gitHead(checkout)
                  pathOk <- if cleanPath.isEmpty then (true: Boolean < (Async & Abort[Throwable])) else (checkout / cleanPath).exists
                yield
                  val drift = head.filter(h => !h.startsWith(sha) && !sha.startsWith(h.take(sha.length))).toSeq.map { h =>
                    Finding(
                      Severity.Warn,
                      "source-commit-drift",
                      where,
                      None,
                      s"source pinned at ${sha.take(8)} but $refsName/$org/$repo is at ${h.take(8)}",
                      Some("re-read the source and update the concept, or leave the pin and accept it is historical")
                    )
                  }
                  val missing =
                    if pathOk then Nil
                    else
                      Seq(Finding(
                        Severity.Warn,
                        "source-path-missing",
                        where,
                        None,
                        s"source path no longer present at the checkout's HEAD: $cleanPath",
                        Some("the file moved or was deleted upstream; the pinned URL still resolves on GitHub")
                      ))
                  Chunk.from(drift ++ missing)
            }
          case _ => (Chunk.empty[Finding]: Chunk[Finding] < (Async & Abort[Throwable]))
      }.map(_.flattenChunk)
    }

  // ---------------------------------------------------------------- rendering

  def renderText(findings: Seq[Finding], verbose: Boolean): String =
    val shown = if verbose then findings else findings.filter(_.severity != Severity.Info)
    val sb = StringBuilder()
    shown.foreach { f =>
      sb ++= f"${f.severity.label}%-5s  ${f.check}%-26s  ${f.location}\n"
      sb ++= s"       ${f.message}\n"
      f.hint.foreach(h => sb ++= s"       hint: $h\n")
    }
    val e = findings.count(_.severity == Severity.Error)
    val w = findings.count(_.severity == Severity.Warn)
    val i = findings.count(_.severity == Severity.Info)
    if shown.isEmpty then sb ++= "no findings\n"
    sb ++= s"\n$e error(s), $w warning(s), $i info"
    if !verbose && i > 0 then sb ++= " (hidden; pass --verbose)"
    sb ++= "\n"
    sb.toString

  def renderJson(findings: Seq[Finding]): String =
    ujson.write(
      ujson.Obj(
        "errors" -> findings.count(_.severity == Severity.Error),
        "warnings" -> findings.count(_.severity == Severity.Warn),
        "infos" -> findings.count(_.severity == Severity.Info),
        "findings" -> ujson.Arr.from(findings.map { f =>
          ujson.Obj(
            "severity" -> f.severity.label,
            "check" -> f.check,
            "path" -> f.path,
            "line" -> f.line.map(l => ujson.Num(l.toDouble)).getOrElse(ujson.Null),
            "message" -> f.message,
            "hint" -> f.hint.map(ujson.Str(_)).getOrElse(ujson.Null)
          )
        })
      ),
      indent = 2
    ) + "\n"
deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall