Skip to content

Systemorph/MeshWeaver / .claude/skills/release

release

Verified

Cut a MeshWeaver release. Two channels, both already wired in .github/workflows. CONTINUOUS = merge to main → multi-arch Docker to ACR, baked and sealed → every install self-updates under its own policy. OFFICIAL = push an annotated v*.*.* tag on a promoted, sealed commit → release.yml PROMOTES that set (retags in ACR + GHCR, release marker, GitHub Release, next-line bump PR). Nothing is rebuilt and nothing ships to NuGet. Use when shipping a release, tagging a version, or wiring/altering the release pipeline. Read BEFORE tagging — a tag is a public, hard-to-reverse publish.

FileKindSize
SKILL.mdskill md28.7 KB
---
name: release
description: Cut a MeshWeaver release. Two channels, both already wired in .github/workflows. CONTINUOUS = merge to main → multi-arch Docker to ACR, baked and sealed → every install self-updates under its own policy. OFFICIAL = push an annotated v*.*.* tag on a promoted, sealed commit → release.yml PROMOTES that set (retags in ACR + GHCR, release marker, GitHub Release, next-line bump PR). Nothing is rebuilt and nothing ships to NuGet. Use when shipping a release, tagging a version, or wiring/altering the release pipeline. Read BEFORE tagging — a tag is a public, hard-to-reverse publish.
user-invocable: true
allowed-tools:
  - Read
  - Bash
  - Grep
  - Edit
---

> The cluster name, resource group and namespace list are deployment identities and live in
> the PRIVATE `Systemorph/Memex` repo — `deployments/aks/envs.json` (cluster + environments)
> and `docs/deployments.md`. Export them once per session rather than hard-coding them here:
>
> ```bash
> eval "$(gh api repos/Systemorph/Memex/contents/deployments/aks/envs.json --jq '.content' | base64 -d \
>   | jq -r '"AKS_RG=\(.cluster.resourceGroup) AKS_CLUSTER=\(.cluster.name) NAMESPACES=\"\([.environments[].ns]|join(" "))\""')"
> ```
>
> Verified to set all three (resource group / cluster / the namespaces) on
> 2026-08-19. Reading it from the source of truth also means a new environment shows up here
> automatically instead of this file going quietly stale.

# /release — ship MeshWeaver (continuous + official channels)

The release pipeline is **tag-driven and already built**. This skill is the runbook for using it
safely; the design rationale lives in
[ReleaseStrategy.md](../../../src/MeshWeaver.Documentation/Data/Architecture/ReleaseStrategy.md)
and the version mechanics in
[ReleaseProcess.md](../../../src/MeshWeaver.Documentation/Data/Architecture/ReleaseProcess.md).

> 🚨 A `v*.*.*` tag fires a **public, hard-to-reverse** publish (ACR + GHCR tags, a GitHub Release).
> Never tag until the commit's CD run has SEALED and the notes page is committed.

## The two channels (don't mix them)

| | CONTINUOUS (steady-state) | OFFICIAL (a release) |
|---|---|---|
| **Trigger** | merge to `main` | push an **annotated** tag `v<major>.<minor>.<patch>` on a promoted, sealed commit |
| **Version** | `3.1.<run#>` — a plain SemVer release per build, the patch is main-cd's `$GITHUB_RUN_NUMBER` (policy `platform-semver-versioning`; retired notation `3.0.0-ci.<run#>` is still READ) | clean `3.1.0` — **the same bytes**, retagged |
| **Workflow** | `main-cd.yml` (after `MeshWeaver Build and Test` passes) | `release.yml` |
| **Docker** | **multi-arch** (`linux-x64;linux-arm64` → OCI image-index) → ACR | ACR retag + GHCR mirror, by digest |
| **Bake / seal** | ✅ platform content, sealed under the compatibility key `c<major>e<epoch>`; core CD seals NO module — the `plugins` publication is sealed by MeshWeaver.Plugins' own lane and never gates the platform | ✅ inherited — `_releases/<clean>` copies the key marker |
| **NuGet** | ❌ never | ❌ **retired** (last publish `3.0.0-rc13`) |
| **Rollout** | an install self-updates onto it ONLY when its `Admin/UpdatePolicy` is `Continuous` **with a pattern** that admits the tag (`3.*`) | every install on the default (`Stable`, clean releases only) self-updates on its next check |

So: **merge to main = build + bake + seal + deploy; tag = promote.** There is no rc line and no
channel word: every main build is a release version `<major>.<minor>.<run>`, and `PlatformVersion`
names the LINE (`3.1.0`). A build that publishes nothing (PR, local) is `<major>.<minor>.0-dev`.

> 📅 **2026-09-12 — what a merge to `main` sets in motion downstream, corrected.** (1) `main-cd.yml`
> has no deploy job: it builds, promotes, bakes, seals and POSTs ONE signed `platform-build` to the
> control instance, then stops; every portal — our own instances included — rolls itself under
> its `Admin/UpdatePolicy` (pull, never push). (2) That build fact no longer wakes the satellites:
> the per-build `meshweaver-framework-released` wave is off by default
> (`Hosting:PlatformBuilds:BroadcastFrameworkReleases`, Plugins#1707) and no satellite lists the
> event. Each node repo rebuilds against the newest SEALED set on its own pushes and once a day —
> that daily run is the full run and is what validates a continuous build; a bundle for a new
> identity therefore arrives with the next daily run, not minutes after the merge. Turning the wave
> on is a MAJOR-bump procedure. Full reference: `Hosting/BuildAndReleaseProcess` (MeshWeaver.Plugins; `get Hosting/BuildAndReleaseProcess` through the mesh MCP).

🚨 **Self-update takes clean releases by default.** A run-numbered build (`3.1.<run>`, or a retired
`3.0.0-ci.<run>`) is rolled onto only under `Continuous` + a `pattern` on `Admin/UpdatePolicy` — a
glob over the tag — and `Continuous` with no pattern IS `Stable` (the poller warns once, naming the
pattern to set). Stable never takes a run-numbered build even though it carries no label
(`VersionSelect.PickTargets`). **Fleet today: every Systemorph-operated instance carries
`Continuous` + `3.*`**, which admits both notations; candidates are ranked by run number. A record
left on the retired `3.0.0-ci*` admits no new build and freezes silently. The `-latest` pointers
(`3-latest`, `3.1-latest`) are a fresh install's STARTING image and never a self-update candidate. Full rule: [ReleaseProcess.md](../../../src/MeshWeaver.Documentation/Data/Architecture/ReleaseProcess.md)
→ "Which build an install takes".

🚨 **THE LADDER — policy `platform-backwards-compatibility`.** Platform builds are backwards
compatible within a compatibility epoch, so platform and plugins roll as SEPARATE steps: the platform
first, keeping the plugin bytes an install runs (no rebuild, no re-seal), then a plugin built against
the RUNNING platform (floor ≤ running, no platform roll). A normal platform build requires NO plugin
seal anywhere — not in CD (`delivery-verdict` judges the platform legs only; core CD has baked no
`plugins` publication since #6143 — MeshWeaver.Plugins' own lane is its only writer), not in the roll selection, not in the sealed-sync
gate. A plugin seal is a precondition ONLY behind a DECLARED break (epoch/major bump in
`src/MeshWeaver.Compiler/platform-compatibility.json`, with a ceiling on the old builds), and then it
is the precondition for that PLUGIN to follow: the platform still rolls (policy
`platform-rolls-independently-of-packages`), each installed plugin with no sealed replacement covering
the target is named as a package advisory on `Admin/UpdatePolicy`, and that plugin keeps its previous
generation until its replacement is sealed.
**Never seal or pin around a break — declare it.** Full procedure: [Deploying Across Platform Versions](../../../src/MeshWeaver.Documentation/Data/Architecture/DeployingAcrossPlatformVersions.md).

🚨 **A platform roll no longer waits for every satellite to re-seal** (2026-09-08,
`Modules:VersionStrictness`, default `Family`). A portal on 3.1.0 adopts a bundle sealed for any
3.x identity whose declared floor is satisfied AND whose platform type references measurably resolve
(`ModulePlatformLink`); one that cannot load compiles from source (or is refused, named, on a
require-prebuilt mesh). `Exact` restores the old identity-only rule; `Minimum` (the Development
default) drops the line check. So "sealed for THIS identity" is no longer the precondition for a
roll — a sealed set of the SAME LINE is. The seal still matters for the sources: a publication sealed
for the portal's own identity brings the repository's sync sources onto its commit
(`IPublicationSyncReconciler`), which is what keeps bundle and sources one unit. Full reference:
[ModuleVersioning.md](../../../src/MeshWeaver.Documentation/Data/Architecture/ModuleVersioning.md)
→ "Version strictness".

## Preconditions for a release (gates the lane enforces — check them before tagging)

### 🚨 Before any of the below: the READINESS gate, which the lane does NOT enforce

**A release may be cut only when NO `sev:B` and NO `sev:H` bug is open, in any of the seven gated
repositories**; `sev:M` and `sev:L` never gate a cut. That is policy
[`release-blocker-gate`](../../../src/MeshWeaver.Documentation/Data/Architecture/PolicyNotProse.md),
which carries when it took effect and who set it — this page carries only the rule.

Enforced by `Governance/Standards/release.cut` in MeshWeaver.Plugins — 7 repositories × 2 labels = 14
`NoOpenIssues` gates — so a release proposal cannot reach `Ready` while one is open. `Memex` and
`MeshWeaver.Feedback` are deliberately outside it. Rubric and the two rules that keep it honest:
[Issue Taxonomy](../../../src/MeshWeaver.Documentation/Data/Architecture/IssueTaxonomy.md).

Check it first — it is the cheapest gate here and it stops everything:

```bash
REPOS="MeshWeaver MeshWeaver.Plugins MeshWeaver.Crm MeshWeaver.SocialMedia
       MeshWeaver.Reinsurance MeshWeaver.Manufacturing MeshWeaver.Education"

# Fail CLOSED first. A label that does not exist answers 0 open issues, which reads as GREEN.
for R in $REPOS; do
  for S in B H; do
    gh api "repos/Systemorph/$R/labels/sev:$S" --jq '.name' >/dev/null 2>&1 \
      || { echo "ABORT: $R carries no sev:$S label — a count against it would read 0 and mean nothing"; exit 1; }
  done
done

for R in $REPOS; do
  for S in B H; do
    printf '%-26s sev:%s = %s\n' "$R" "$S" \
      "$(gh api "repos/Systemorph/$R/issues?state=open&labels=sev:$S&per_page=100" \
           --jq '[.[]|select(.pull_request==null)]|length')"
  done
done
```

🚨 **Read a zero against its coverage — THREE ways this count lies, all measured.** (1) A
`search/*` query answers 0 when it is truncated, rate-limited (search has its OWN ~30/min limit) or
unanchored, indistinguishable from "none open" — which is why the loop uses the REST issues endpoint
instead. (2) PRs share that endpoint, hence `pull_request == null`. (3) 🚨 **An unknown label folds the
gate to GREEN**: `labels=sev:DOESNOTEXIST` answers `0`, byte-identical to a real label with nothing
open, so a renamed or missing label ships a known blocker silently. The assertion above is what makes
the count mean something — and it uses the SINGLE-label endpoint deliberately, because **listing
labels is the trap one level down**: core holds 13 pages of them and its `sev:*` four sit on page 2,
so a one-page existence check reports them absent from the repository that carries the most open
`sev:H`. `repos/{o}/{r}/labels/sev:H` needs no pagination and 404s when the label is gone.

🚨 **A green gate reached by DOWNGRADING an `sev:H` to `sev:M` is not a green gate.** That is the
cheapest defeat of this bar and `release.cut`'s acceptance criterion names it; spot-check the most
recently re-labelled two.

1. **The commit is on `main` and its CD run SEALED**: `Promote`, `Verify every image shipped` and
   `Bake platform content in the shipped image + publish` all `success` — read the JOBS, never the
   run's conclusion. (Core CD has had no `Plugins: bake + seal` job since #6143; the `plugins`
   publication `release.yml` also asserts is sealed by MeshWeaver.Plugins' own lane.)
   ```bash
   gh api "repos/Systemorph/MeshWeaver/actions/runs?head_sha=<sha>" \
     --jq '.workflow_runs[] | select(.name=="Continuous Delivery (main)") | .id' | head -1 \
     | xargs -I{} gh api "repos/Systemorph/MeshWeaver/actions/runs/{}/jobs?per_page=100" \
       --jq '.jobs[] | select(.name | test("Promote|Verify every|Bake platform content")) | "\(.name): \(.conclusion)"'
   ```
   🚨 **Name the target set only AFTER this reads success, and state the requirement as a COMMIT until
   then** ("the first sealed set cut from `<sha>` or later"). A set's name carries its RUN number, and
   the one-pending-slot rule cancels the pending run on every merge to `main` while the commit stays
   valid — 7910 → 7911 → 7912 inside 20 minutes on 2026-09-06. Send a set's name and its digests in
   ONE message. Full reference: [ContinuousDeliveryContract](../../../src/MeshWeaver.Documentation/Data/Architecture/ContinuousDeliveryContract.md) → "A set is named by its RUN".
2. **`PlatformVersion` at that commit equals the tag** (`3.0.0` ↔ `v3.0.0`). The lane refuses a
   mismatch; so does it refuse a `-rc`/`-beta` suffix and a lightweight tag.
3. **The notes page exists at that commit**: `src/MeshWeaver.Documentation/Data/ReleaseNotes/3_0_0.md` (a
   FLAT file — a `3_0_0/index.md` is embedded as `_3_0_0` and served at the wrong path)
   (`Category: Release Notes`). The GitHub Release is published FROM it.
4. **Secrets present** (can't be read; confirm with the operator): `AZURE_CLIENT_ID`/`AZURE_TENANT_ID`/
   `AZURE_SUBSCRIPTION_ID` (OIDC → ACR + the artifact stores, via the `release` environment),
   `MESHWEAVER_APP_ID`/`MESHWEAVER_APP_PRIVATE_KEY` (the next-line PR), repo var
   `BAKE_PUBLISH_TARGETS`. The lane's `preflight` job fails RED naming any that is missing.
5. **The OWASP ZAP scans ran against the candidate and every finding has a disposition on the
   notes page** — the lane cannot check this one; it is the operator's gate (next section).

## Security scan — every release, before the tag

Both runs, against the deployment that serves the candidate build. Full reference, verdict rules
and the findings each release carried:
[SecurityScanning.md](../../../src/MeshWeaver.Documentation/Data/Architecture/SecurityScanning.md).

```bash
ZAP=ghcr.io/zaproxy/zaproxy:2.17.0          # pinned by VERSION; the version goes on the notes page; bump deliberately
OUT=~/.cache/zap-scan-$(date -u +%F); mkdir -p "$OUT/public" "$OUT/auth"   # a PLAIN dir: Docker mounts it
# Each run's console output IS its log (the verdict is its last line), so it is captured; the
# scripts exit 0 PASS / 2 WARN / 1 FAIL / 3 scanner error — echo it, never let it end the shell.
# 1. PUBLIC, ACTIVE — anonymous, safe against production
docker run --rm -v "$OUT/public":/zap/wrk/:rw -t "$ZAP" \
  zap-full-scan.py -t https://portal.example.com -r public-full.html -J public-full.json -w public-full.md \
  > "$OUT/public/public-full.log" 2>&1; echo "public exit=$?"
# 2. AUTHENTICATED, PASSIVE — never zap-full-scan with a session on production (it fires payloads
#    AS THE USER at every write endpoint). $COOKIE = the full Cookie header of a real browser
#    session (.AspNetCore.Cookies; an API token does not authenticate the SPA). -j = AJAX spider,
#    the only thing that reaches the bundles a signed-in page loads.
docker run --rm -v "$OUT/auth":/zap/wrk/:rw -t "$ZAP" \
  zap-baseline.py -t https://portal.example.com -j -r auth-report.html -J auth-report.json -w auth-report.md \
  -z "-config replacer.full_list(0).description=sess -config replacer.full_list(0).enabled=true -config replacer.full_list(0).matchtype=REQ_HEADER -config replacer.full_list(0).matchstr=Cookie -config replacer.full_list(0).regex=false -config replacer.full_list(0).replacement=$COOKIE" \
  > "$OUT/auth/auth-baseline.log" 2>&1; echo "authenticated exit=$?"
# 3. VERDICT — the last line of each captured log; FAIL-NEW must be 0 on both, and rule 10003 blocks:
tail -1 "$OUT/public/public-full.log" "$OUT/auth/auth-baseline.log"
grep -h "Vulnerable JS Library" "$OUT"/*/*.log        # PASS = clean; WARN-NEW = a shipped library with advisories → blocker
```

Then write the two verdict lines and a disposition for EVERY `WARN-NEW` — fixed, or carried with
the rule id, the reason and the tracking issue — on the notes page (the 3.0.0 page's *Security*
section is the shape). A WARN without a disposition is an untriaged finding, not a pass. The raw
reports stay under `$OUT` (they are a site map with bodies); the cookie is a live session — keep it
out of every repo and sign out when done. 🚨 The version string on a package manifest is not the
verdict: the DOMPurify finding on 3.0.0 (MeshWeaver#3378) lived inside a NuGet package's static
bundle, invisible to Dependabot and to any source build — only the served bytes tell.

## Continuous release (the common case — just merge)

```bash
# 1. Confirm CI green on the merge commit (NEVER ship a red commit — main-cd gates on this).
gh api "repos/Systemorph/MeshWeaver/actions/runs?branch=main&per_page=3" \
  --jq '.workflow_runs[] | "\(.name) \(.head_sha[0:9]) \(.status) \(.conclusion)"'
# 2. Merge the PR. main-cd.yml then fires automatically on the green test run:
#    builds multi-arch portal-ai + migration + mw-plugin-test, promotes <version>;<sha>;main to ACR,
#    bakes + seals the platform content ONLY, and notifies the control instance. The Plugins
#    modules are packed, baked and sealed by MeshWeaver.Plugins on its own lane (#6143); core's
#    `published-modules` job only reads that publication.
# 3. Every OTHER install whose Admin/UpdatePolicy is Continuous + a pattern admitting the tag
#    (3.*) self-updates from ACR on the next publication EVENT (no action). Stable installs
#    (the default) wait for the clean release.
```

## 🚨 A merged fix can look SHIPPED while producing no image — verify the IMAGE, never the tick

**Publishes are BATCHED (maintainer, 2026-08-16).** When repo var `CD_BATCH_WINDOW_MINUTES` is set,
a green merge whose newest published set is younger than the window does NOT publish — the hourly
reconciler publishes main's tip on its next tick instead. The decision step's summary says
`🕐 Batched` when this happened, so "merged + green + no new tag" inside the window is INTENTIONAL,
not a failure. To ship a specific fix immediately: `gh workflow run main-cd.yml --ref main` (the
manual path bypasses the window by construction). The probe fails OPEN — an unreadable registry
publishes rather than blocks.

🚨 **The window is NOT the wait.** A batched merge leaves main's HEAD without an image set, which is
exactly what the reconciler heals — so it ships on the next TICK, whatever the window says. The
window controls how many merges ride one image; the **reconcile cadence controls the tail**. Now:
window 60 min, one tick at `:23`, so the worst case is ~1 h plus the ~20 min build.

🚨 **Publication frequency is NOT portal-restart frequency — but only because a floor now exists.**
Since #1773 an install checks per publication *event*, and `UpdatePolicyKind` picks a channel, not a
cadence. #1780 added **`SelfUpdate__MinRollInterval`**, a restart budget set to `1h` in
`values.aks.yaml`, deliberately matched to the tick. **If you change one, change the other.**

🚨 **Promoted is not SEALED.** `Promote` and `Verify every image shipped` can be green while the
platform bake failed, or while MeshWeaver.Plugins' own lane has not yet sealed the `plugins`
publication for that identity (core CD no longer seals it — #6143) — the images exist, and a tag on
that commit is refused by `release.yml`'s `check-release-availability.sh`. The self-updater does NOT hold the platform roll for it (policy
`platform-rolls-independently-of-packages`): it rolls, and names each package without a sealed bake as
an advisory on `Admin/UpdatePolicy` (`… no sealed content bake …`); those packages boot-compile or keep
their previous generation. A `heldReason` of that shape on an old record is history, not a hold.

🚨 **But a seal never holds CONTENT SOURCES any more** (policy `module-sync-per-manifest-hash`). An
instance syncs every module it has, each judged alone by the content hash in its `manifest.lock`:
unchanged is a no-op, changed syncs to the incoming commit, and a module whose declared platform
floor is above the running platform is declined by name without holding its siblings. The seal
decides only whether a NodeType ADOPTS prebuilt bytes or COMPILES from the synced source. A
content-repo merge that "did not arrive" is therefore not waiting for a seal: read the module's
outcome on `/health` (`publication-seal`) instead. Manual:
[ModuleSyncPerManifestHash.md](../../../src/MeshWeaver.Documentation/Data/Architecture/ModuleSyncPerManifestHash.md).

### The `main-cd.yml` trigger gate and its three SILENT consequences

`main-cd.yml`'s `workflow_run` path is gated on `event == 'push' && head_branch == 'main'` — that
gate is what stops a **fork's** pull_request run (whose `head_branch` can also be "main") from
publishing untrusted code with this repo's secrets, so never relax it. Three consequences trip
people up, all silent:

1. **No Build-and-Test run on main at all.** CD reacts to that workflow completing; if it never ran
   on the merge commit (a CI incident, a stalled queue), CD sits `SKIPPED` with nothing to react to.
2. **`workflow_dispatch` of Build-and-Test can never ship.** It RUNS and genuinely tests the merge
   commit, so main shows a **green Build-and-Test** — but its `event` is `workflow_dispatch`, not
   `push`, so CD still skips.
3. **A cancelled main run publishes nothing.** CD's delivery gate keys on `Consolidate test results`
   reaching `success` **for that SHA**. This is why runs on `main` are never cancelled.

**None of these is terminal.** CD carries a **reconciler**: an hourly `schedule` (plus its own
`workflow_dispatch`) resolves main's tip through the API, asks ACR whether that commit has the
complete image set, and publishes only when it does not — bounded at 3 attempts per commit.

- **To kick CD by hand: `gh workflow run main-cd.yml --ref main`.**
- It heals **HEAD, not the commit that failed** — deliberately: re-publishing older code would mint
  a higher run number for it and roll every install *backwards*.
- **Publication is all-or-nothing**: each leg pushes only a non-selectable `staging-<sha>-<run_id>`
  tag, and the `promote` job applies the real tags only after every leg succeeds, ending with
  `memex-portal-ai:<version>` — the single write the self-updater acts on.

**Before believing something is deployed, check the IMAGE, never the green tick** — the Fleet
Console (`/Hosting/Console`) shows the RUNNING version per instance; the cluster read is the
break-glass form:

```bash
az acr repository show-tags -n meshweaver --repository memex-portal-ai --orderby time_desc --top 5 -o tsv
az aks command invoke -g "$AKS_RG" -n "$AKS_CLUSTER" --command \
  "kubectl get deploy -A -o custom-columns=NS:.metadata.namespace,IMAGE:.spec.template.spec.containers[0].image --no-headers | grep memex-portal-ai"
.github/scripts/check-image-set.sh <short-sha>   # the exact assertion CD itself makes (the pair-tag argument is retired)
```

## Official release (promote a sealed continuous build)

```bash
# 0. Pick the commit: on main, its CD run SEALED (preconditions above), PlatformVersion == the tag.
grep -m1 '<PlatformVersion Condition' Directory.Build.props          # e.g. 3.1.0 — use THIS value below
# 1. Make sure the notes page is committed at that commit — WITH the Security section: both OWASP
#    ZAP verdict lines and a disposition per WARN (section above):
#    src/MeshWeaver.Documentation/Data/ReleaseNotes/3_1_0.md   (flat — never <version>/index.md)
# 2. Tag it ANNOTATED and push — this is the whole release (maintainer: only once every open
#    issue is closed). The tag MUST equal PlatformVersion at <sha>; release.yml refuses otherwise:
git tag -a v3.1.0 -m "MeshWeaver 3.1.0" <sha> && git push origin v3.1.0
#    → release.yml: resolves the 3.1.<run> build for <sha> → asserts complete + sealed → writes
#      _releases/3.1.0 → retags memex-migration, mw-plugin-test, memex-portal-ai (last) → mirrors
#      to GHCR (no line pointer moves: CD already moved them) → publishes the GitHub Release →
#      opens "release: PlatformVersion 3.1.0 → 3.2.0".
# 3. Merge that bump PR. Main builds keep sorting ABOVE the release either way (3.1.<run> > 3.1.0);
#    the bump only lets Stable see the NEXT clean release.
gh api "repos/Systemorph/MeshWeaver/pulls?state=open&head=Systemorph:release/open-3.2.0-line" --jq '.[].html_url'
```

What the lane REFUSES, each with a red step naming the fix: a non-clean version (`v3.0.0-rc14`),
a lightweight tag, a commit not on `main`, a `PlatformVersion` mismatch, a commit `main-cd` never
promoted, a set whose bake is not sealed, and a release with no notes page.

## "All portals update" — how (and how to confirm)

Two mechanisms, both live:
- **Push (CD):** `main-cd.yml`'s `deploy` matrix rolled our own portals directly.
  *📅 2026-09-12: no longer — `main-cd.yml` carries no `deploy` job; our own portals roll by
  the pull mechanism below like every other install (their overlays in the deployments repository carry
  a floor image, never a pin, applied by that repo's `helm-release.yml`). See
  `Hosting/BuildAndReleaseProcess` (MeshWeaver.Plugins; `get Hosting/BuildAndReleaseProcess` through the mesh MCP).*
- **Pull (self-update):** `SelfUpdateHostedService` runs on EVERY install. It reads
  `Admin/UpdatePolicy` (default **Stable** — clean tags only, i.e. what `release.yml` promoted),
  lists ACR tags, takes the newest ADMITTED one (`VersionSelect.SelectCandidates` +
  `RollSelection`), asks `ReleaseAvailability` once about the installed packages and reports what it
  refuses as advisories, never a reason to take an older tag (policy
  `platform-rolls-independently-of-packages`), then patches its own Deployment in-pod. `Continuous` + `pattern` (e.g. `3.*`) makes that major's run-numbered builds
  eligible; `Continuous` without a pattern is `Stable`.

Confirm a roll-out — the RUNNING version per instance is on the Fleet Console (`/Hosting/Console`
on the instance that hosts the fleet records), and a roll you order by hand is a `Roll` `Hosting/InstanceAction` with
the tag, never `kubectl set image` (policy:
[OperatingFromThePortal.md](../../../src/MeshWeaver.Documentation/Data/Architecture/OperatingFromThePortal.md)).
The cluster reads below are the break-glass form of the same three questions:
```bash
# ACR has the new tag:
az acr repository show-tags -n meshweaver --repository memex-portal-ai -o tsv | tail
# Each portal serves + runs the new image (break-glass: private cluster → az aks command invoke):
az aks command invoke -g "$AKS_RG" -n "$AKS_CLUSTER" --command \
  "kubectl -n <ns> get deploy memex-portal-deployment -o jsonpath='{.spec.template.spec.containers[0].image}'"
# The release's identity marker (what a Stable install's gate reads):
.github/scripts/check-release-availability.sh 3.0.0 meshweaver-content plugins   # needs BAKE_PUBLISH_TARGETS + az login
```

## Verify a release is healthy (before declaring done)

🚨 **`/api/version` reports the CD run's RESOLVED target commit**, which can differ from the run's
`head_sha` when two merges land in one queue group (2026-09-09: set 8131 = run head `e1039813e`,
`/api/version` = `3853374f7`, one second earlier). To name the set a portal runs, match the commit
against the run's "Resolve the target commit" job output — never against the run head or `main`.

- Migration log shows `Database migration completed. Version: N` AND the portal serves HTTP 200
  (see [DeploymentAKS.md](../../../src/MeshWeaver.Documentation/Data/Architecture/DeploymentAKS.md)).
- The self-updater logged its decision (picked newer / already current / held with a reason), not
  a 403 (missing AcrPull workload-identity grant — armed path no-ops with a logged 403, never crashes).
- `Admin/UpdatePolicy.heldReason` is empty: a hold names an unsealed bundle, and that is a delivery
  incident, not a release.

🚨 **And the running IMAGE is still not the served ANSWER.** *"The grain keeps serving old state
until we send a dispose request"* (maintainer, 2026-09-17): a per-node hub binds its configuration
once at activation and is pinned by address. A roll ends the activations on the pods it replaces, so
the gap is not the roll itself — it is everything a portal absorbs **while it keeps running** (a
package published or installed, a NodeType recompiled in place) and what a fresh activation then
chooses. Whether an instance converges on a newly published build is `Modules:AutoRecycleOnStaleBuild`,
and the code default (**off** — only a banner, so viewers must click) is NOT what the AKS chart sets
(**true**, fleet-wide): read the effective value rather than either. Exercise the feature at the
running address; recycle the addresses that still answer the old way and name them. A recycle makes
the activation re-read, it does not decide what the re-read finds — so a second one that answers the
same is a delivery question, not an activation one. Full reference:
[StaleStateUntilRecycle.md](../../../src/MeshWeaver.Documentation/Data/Architecture/StaleStateUntilRecycle.md).

## Pipeline files (edit here to change the pipeline)

- `.github/workflows/main-cd.yml` — continuous: multi-arch build + ACR promote + bake/seal + CD deploy.
- `.github/workflows/release.yml` — official: promote a sealed set on a `v*.*.*` tag; GitHub Release;
  next-line bump PR.
- `.github/workflows/base-image-acr.yml` — the hand-authored multi-arch base, on demand.
- `.github/scripts/check-image-set.sh` / `check-release-availability.sh` / `publish-bake-bundles.sh`
  — the set, the seal, the marker; `release.yml` reuses the first two verbatim.
- `Directory.Build.props` — `PlatformVersion` (the line) + the `<major>.<minor>.<run>` composition from `-p:PlatformBuildNumber` (main-cd only).
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall