Systemorph/MeshWeaver / .claude/skills/deployment
deploy
VerifiedDeploy MeshWeaver — the two routes (shared AKS cluster, and Azure Container Apps via the Aspire test/prod modes) and the traps that make a deploy look done when it is not. Use when rolling a code change onto a portal, standing up or patching an environment, or verifying that what is running is what you built. FIRST RULE (maintainer, 2026-09-08): on AKS an operation is a Hosting/InstanceAction on the control instance (Roll, Restart, Suspend, Audit, Reconcile, …) executed by the in-cluster operator — az aks command invoke / kubectl are break-glass, never the procedure. Also covers the bootstrap build-image / set-image / restart sequence the operator runs for you, why an env deploy.sh is first-time setup and not a code-update path, why the database migration is a run-once Job that a helm upgrade runs — never a Deployment to roll — and the DbVersionGate that makes a portal refuse to serve ahead of its schema.
/deploy — pick the route by TARGET, then verify what is actually running
🚨 The cluster is not the surface — the control instance is (maintainer, 2026-09-08: "all the
operations through the memex api", "no direct access of aks"). On AKS, an instance is a
Deployments/<name> record on the CONTROL INSTANCE (reached through that instance's own MCP server;
any other portal's Deployments/* copy is a lagging second sync, not the record) and every operation is a
Hosting/InstanceAction node — Roll (the record's image pin, or an explicit imageTag, then WAIT
for the rollout), Restart, Suspend/Reactivate, Audit, Reconcile, HelmRelease — run by
the in-cluster operator and reported on the same node. Every az aks command invoke / kubectl
line below is what the operator runs FOR you, kept here as the bootstrap form and as break-glass;
a break-glass write is half an operation (the Hosting manual's "Break glass" section lists the
other half). Policy page, with what the API does NOT answer yet:
OperatingFromThePortal.md.
Two deploy routes, different targets — neither deprecated. Don't mix them.
- AKS — a portal instance on the shared cluster. Full ref: DeploymentAKS.md.
- Azure Container Apps — the Aspire
test/prodmodes, via MeshWeaver.Pluginstools/deploy.sh prod|test. Full ref: DeploymentContainerApps.md.
Index: Deployment.md.
🚨 Before any AKS deploy, read DeploymentAKS.md end-to-end — it is the source of truth for
build → roll-out → verify. 🚨 It no longer describes an auto-baked mesh-local #r package feed: the
BakeMeshLocalFeed target was REMOVED (#395), dist/packages is gone, and a legacy
#r "nuget:MeshWeaver.BusinessRules.Generator" now hard-fails on a deployed image ("The local source '/app/dist/packages' doesn't exist" — the prod BalanceSheet failure). The generator ships
built-in and GeneratorPipeline strips that #r instead. Nothing bakes at BUILD time on any
machine, dev Macs included: --bake-output exists only in CI scripts, never in a
.targets/.props/.csproj. The commands below are a quick reference, not a substitute for the
doc.
🚨 The ladder: which image and which plugins, and when a roll is HELD
Policy platform-backwards-compatibility: a portal rolls to the newest promoted image its update
policy admits and on which no installed plugin is held — and on a normal build nothing is held, with
NOTHING sealed for the new build. It keeps its plugin bytes across the roll, and adopts a newer plugin
publication later, independently, when that publication's floor is ≤ the running platform. The one
roll hold is a DECLARED break (another compatibility key, or a ceiling an installed build declares)
with no replacement sealed for the target: the self-updater's candidate is declined as
PlatformRangeExceeded, naming the plugin and both versions, and the portal lands on the newest
release its plugins still cover. A routed Roll runs unattended under a Continuous record whose
pattern admits the tag; it carries the target's migration Job first. 🚨 Never pin an image tag in a
record or a chart value — not to unblock a roll, not ever (PlatformDeliveryNeverWaitsOnPluginsGuard
refuses one under deploy/). Full procedure and decision table: Deploying Across Platform Versions.
🚨 First: the memex API, not the cluster
Maintainer directive, 2026-09-08: every operation goes through the memex API — no direct az /
kubectl access. The control instance (e.g. https://control.example.com, reached through its own MCP server, module
Hosting from MeshWeaver.Plugins) exposes each of the recipes below as a Hosting/InstanceAction node whose
answer is a node; the in-cluster operator and the monitoring stack hold the only credentials.
Reach for the commands further down ONLY when the control plane itself is what is broken (the
"break glass" section of the fleet guide, get @Hosting/Guide).
| you want | create (MCP create, or POST the node) | read the answer at |
|---|---|---|
roll <id> onto a tag | pin pinnedImageTag on Deployments/<id>, then { "requestedAction": "Reconcile", "confirmation": "<id>" } — or { "requestedAction": "Roll", "imageTag": "<tag>", "confirmation": "<id>" } for the portal image alone | the action node's state/log; then a Sample |
what is it running, per replica (image, ready, restarts, started, its own /health + detail) | { "requestedAction": "Sample" } | Ops/Status/<id> (replicas[], warnings[]), the Deployment page's status strip |
| the last hour's error lines | { "requestedAction": "Logs", "query": "fail:|crit:|Exception", "sinceMinutes": 60, "limit": 300 } | the action node's logQl / entryCount / truncated; lines under Ops/Logs, @Deployments/<id>/area/Logs |
| what lives only on the cluster | { "requestedAction": "Audit" } | Ops/Audit/<id> |
| re-read ONE address no person may recycle (a NodeType and its dependency network, or any node) | { "requestedAction": "Recycle", "recycleTarget": "<path>", "reason": "…" } + one approval | the run's log. Manual: Hosting/RecycleAction (Plugins) |
| delete a space no person may delete (owner gone, stranded partition) — the sanctioned break-glass for DATA, since a global admin is not a data superuser | { "requestedAction": "DeleteSpace", … }; it parks WITH the full plan | the plan on the node, the audit record. Manual: Hosting/DeleteSpaceAction (Plugins) |
| upgrade the cluster's Kubernetes version (control plane + every pool) | plan: { "requestedAction": "UpgradeCluster", "dryRun": true }; then { "requestedAction": "UpgradeCluster", "upgradePlan": "<plan node path>", "confirmation": "<id>" } + ONE mesh approval — never az aks upgrade (policy cluster-upgrade-governed) | the plan on the dry-run node; the run's log, aks_versions; a stuck portal roll refuses it by name. Manual: Hosting/ClusterUpgrade (Plugins) · DeploymentAKS.md "Kubernetes version upgrades" |
All nodes are nodeType: Hosting/InstanceAction, content.$type: InstanceActionContent,
content.deployment: "Deployments/<id>". The full table — including what each action does and
which credential runs it — is the fleet guide's "Roll, restart, observe — the ops actions".
🚨 A gated action: you approve a PLAN, and the approval is bound to it
Every mutating kind parks at awaiting approval WITH the plan it will run (the steps and commands, in the order they run). The approval binds the request, the action, the deployment, the image tag, the kind's own binding (a Recycle's target and force flag, a DeleteSpace's space) and the DIGEST of that plan. A plan that would come out differently re-parks; it never runs unapproved. What else is true, and bites:
- Every step runs as system, and the plan PRE-FLIGHTS rights and scale (policy
governed-action-preflight, proposed; implementation in flight). An action refuses at park when a step cannot complete, never partway through a destructive run. The evidence is a DeleteSpace that failed at step 9 after it had already removed the space's grant and GitSync. It hit 31,138 per-node delete validations against a 25 s bound. Until the pre-flight ships, read a destructive plan's scale (the rows per table) yourself before you approve it. - Plans are deterministic. A clock-derived value (a dump's object name) comes from
planSeedAt, stamped at the first park of an approval round. A node that re-parks every few seconds with a movingparkedAtis a plan that is not deterministic: a defect to file, not an approval to retry. - The requester is recorded once, at the first park. A re-park never rewrites it, so the approver is never mistaken for the requester.
- Approve with the button. It attests the AUTHENTICATED clicker (an HMAC over the approval, the request, the park and the node revision). An approval typed by someone else is not an approval.
- Only the configured maintainer (
Hosting:Operator:Maintainer) may approve their own request. From an agent session that is ONE MCPpatch, written as the maintainer, carryingrequestedBy,approvedBy,approvedAtand every binding field as parked (approvedRequest,approvedAction,approvedDeployment,approvedImageTag, plusapprovedRecycle/approvedPlanwhere the kind binds them). Create the action and WAIT FOR THE PARK first: an approval in the create write, or one dated beforeparkedAt, is refused and re-parks.approvedPlanis not stored on the node, so recomputeActionPlanSnapshot.Digest()from the parkedactionPlan(sha256:lower-hex). Split the approval into several writes, or let a binding field drift, and the gate re-parks. This is a write in the maintainer's name, so do it only when the maintainer asked for it. - The digest binds the COMMANDS, not the values file or the operator environment rendered at dispatch. After a record change that follows an approval, re-request the action rather than trusting the binding to catch it.
- Approvable is not yet runnable. An approved plan can still stop on an unbound
$VARor a declared vault object that is absent. Check both before you approve. - A
Runningaction survives a restart of the control instance. The watcher re-opens every orphanedRunningaction on start and resumes following it. Do not re-file it. - One rollout budget. How long a roll may take is the chart's own progress deadline
(
ChartRolloutBudget), read by the action's wait and by the stuck-roll alarm alike. - A
Reconcilethat ADDS a Key Vault key needs aRestartafter it (open finding, Systemorph/MeshWeaver.Feedback#48): the CSI driver syncs the new object after the pod has resolved its environment. - A
Reconcilecan refuse on the Key Vault values half before helm runs, if the half carries structure. Run aHelmReleaseaction withhelmAction: capturefirst, then the Reconcile. - A secret is never set or read with
azby hand (policysecrets-write-only-entry). Paste or generate it on the control instance:Deployments/<id>→ Set Key Vault secrets…. That files aSetSecretsaction whose Job writes as the WRITER identity (it can list, set and delete, but never read a value) and stamps themw-fpfingerprint tag. Compare the two ends of a pairing by that fingerprint, never by reading the values. CI refuses a new hand-run vault command (check-manual-keyvault.py). Full model: SecretsWriteOnlyEntry.md.
The rules, the page layout and the threat model: Hosting/AksOperationsViaActions (Plugins,
"Approval in the mesh" and "What operating it taught") · DeploymentAKS.md
"Diagnostics — through the memex API first".
🚨 Authoring or reading a governed action: system credentials from the first step, pre-flight at park
Policy governed-action-preflight. Every step of a governed action runs under the system
identity from its first step, and at PLAN time — before it parks for approval — the action
verifies each step is feasible: the rights the step needs as system, and the scale bound of the
mechanism it uses. A step the system identity cannot perform, or one known to exceed its bound, is
refused at park with the reason — never discovered partway through a destructive run, where it
leaves the target half-removed. The plan names, per step, the identity, the mechanism and its bound.
The incident that set it: a governed DeleteSpace of UWDeepfield (31,138 descendants) ran its
grant and GitSync removals as system, then failed in Delete content — a per-node recursive
DeleteNodeRequest whose ValidateDeleteRequest pre-flight fan-out could not settle in its 25 s
bound — leaving the space with no grant and no sync but all its content. A whole-space deletion now
drops the partition as ONE teardown (PartitionTeardown.TearDownPartition, which itself refuses any
identity but system), and its plan refuses at park when a per-node leg exceeds its bound. Manual:
Partition Teardown →
The direct teardown; the action's own manual is Hosting/DeleteSpaceAction (MeshWeaver.Plugins).
🚨 Probes: the startup budget covers a cold boot, never a bake
The bake gate and required_modules are ROLL GATES (policies bake-gate-readiness-only,
required-modules-readiness-only): their verdict is read by /ready alone, so a refusal stalls the
roll with the previous image serving and kills nothing. The startup probe proves only that the
process booted, and the fleet's portal instances run it at failureThreshold: 60 (600 s). Never raise
probes.startup to "cover a bake" — that is the configuration that killed every container of both
images every 3 h and took the control instance down. A container that dies at exactly
periodSeconds × failureThreshold after boot is a startup-probe kill, not a crash: read Sample's
lastTerminationReason / lastExitCode per replica. Full account and incident history:
TheBakeGateOnlyStallsARoll.md.
🚨 A schema change is planned, and every roll migrates first
Policy db-migration-planned: migrations are expand-only, a DbVersion bump declares
Db-migration: V<N> — … in its PR, every release publishes ExpectedDbVersion, and every roll path
runs the migration Job before the image moves or refuses (the operator's run.sh interlock does it
whatever Hosting generation planned the roll). The one exception is the break-glass HelmRelease
dispatch; use Reconcile. Manual:
PlanningADatabaseMigration.md.
The AKS route (break glass — the control plane is what is broken)
🚨 Pass --subscription <subscription> on EVERY az call, as every example below does.
<subscription> is the fleet's own subscription; its id is in the estate's private deployments repository
(see "Where an estate's instances are declared" in
Instances.md), never in this public one. The signed-in account's default subscription may
belong to another tenant's installation, which holds another cluster with the SAME name
(<aks-cluster>); without the flag you read the wrong system and nothing says so.
az role assignment delete --ids … needs it too, or it fails with InvalidAuthenticationTokenTenant.
A portal instance runs on the shared AKS cluster <aks-cluster> (RG <aks-resource-group>,
swedencentral) — namespace <namespace> — against the Postgres Flexible Server, images in ACR
meshweaver.azurecr.io. Private cluster: kubectl ONLY via
az aks command invoke --subscription <subscription> -g <aks-resource-group> -n <aks-cluster> --command "…" — and only when
the API above cannot answer.
On AKS a code update = the record's image pin + a Roll action. What the operator then runs is
build image → set image → restart (the AKS route does NOT use tools/deploy.sh or aspire deploy
— those are the Container Apps route). The commands, for the bootstrap / break-glass case only:
az acr login --subscription <subscription> -n meshweaver
# Portal (custom base) AND migration (the migration is what creates schema + the matview):
dotnet publish ../MeshWeaver.Plugins/src/Memex.Portal.Distributed/Memex.Portal.Distributed.csproj -c Release \
-t:PublishContainer -p:ContainerRegistry=meshweaver.azurecr.io \
-p:ContainerRepository=memex-portal-ai -p:ContainerImageTag=<tag> \
-p:ContainerBaseImage=meshweaver.azurecr.io/memex-portal-ai-base:latest
dotnet publish memex/aspire/Memex.Database.Migration/Memex.Database.Migration.csproj -c Release \
-t:PublishContainer -p:ContainerRegistry=meshweaver.azurecr.io \
-p:ContainerRepository=memex-migration -p:ContainerImageTag=<tag>
# Roll out (NS = the instance's namespace). 🚨 The MIGRATION is a Job, not a Deployment — see below.
az aks command invoke --subscription <subscription> -g <aks-resource-group> -n <aks-cluster> --command "\
kubectl -n <NS> set image deployment/memex-portal-deployment memex-portal=meshweaver.azurecr.io/memex-portal-ai:<tag>; \
kubectl -n <NS> rollout restart deployment/memex-portal-deployment; \
kubectl -n <NS> rollout status deployment/memex-portal-deployment --timeout=300s"
- An env's
deploy.shis first-time ENV SETUP only (helm install + PVCs + KV SecretProviderClass + ingress + connection-string patch). Do NOT use it for a code update — it re-runs the whole chart and can reset live config. 🚨 Env folders live in the estate's PRIVATE deployments repository, notdeploy/aks/envs/<env>/— they moved out 2026-08-08/09 (a69959165) because their directory names are tenant identities;deployments/aks/example/(Systemorph/Memex, moved from this repo'sdeploy/aks/envs/example/) in this repo is the reference template only. - Don't run
tools/deploy.shoraspire deployagainst the AKS cluster — those are the Container Apps route (a different target), not a code-update path for AKS.
🚨 The migration is a run-once Job, NOT a Deployment
deploy/helm/templates/memex-migration/job.yaml: kind: Job, restartPolicy: Never, the name
embeds .Release.Revision, ttlSecondsAfterFinished. It runs on helm upgrade, not on an
image-tag roll — so a kubectl set image / rollout restart aimed at a
memex-migration-deployment is a command with no object: the chart does not define one, and any
that exists in a live namespace is a cluster-only orphan of a chart revision that predates the Job
(#1788). Do not resurrect it. A MigrationWorkloadModelGuard
(test/MeshWeaver.Documentation.Test) fails the build if such a command returns to AGENTS.md, a
skill under .claude/skills/, a doc, or a deploy script.
A crash-looping migration pod is a FAILURE, not noise. It used to be modelled as a Deployment,
which restarted it forever after each clean exit — three prod namespaces sat at 50/53/38 restarts,
each rebuilding public.top_level_index across every partition schema. Documenting that as
"benign" is what made a real migration failure unreadable. With the Job form the pod runs once and
stops, so CrashLoopBackOff on it now means exactly what it says.
Before declaring a deploy successful, confirm the Job's log shows
Database migration completed. Version: N AND the portal serves (HTTP 200):
az aks command invoke --subscription <subscription> -g <aks-resource-group> -n <aks-cluster> --command \
"kubectl -n <NS> get jobs -l app.kubernetes.io/component=memex-migration"
az aks command invoke --subscription <subscription> -g <aks-resource-group> -n <aks-cluster> --command \
"kubectl -n <NS> logs job/memex-migration-<revision>"
The schema has a hard gate. DbVersionGate (a hosted service in Memex.Portal.Distributed)
reads admin.mesh_nodes.db_version at startup and StopApplication()s with a LogCritical when it
is below DbVersion.Latest. So a portal rolled ahead of its schema refuses to serve rather than
serving a half-migrated database — loud, and the reason the ordering between the two workloads
cannot silently invert.
Verify the IMAGE, never the green tick
A merged, green, CD-published change is not a deployed change until the running pod says so. The delivery chain, the publish batching window and the reconciler are in /release; the assertions are:
az acr repository show-tags --subscription <subscription> -n meshweaver --repository memex-portal-ai --orderby time_desc --top 5 -o tsv
az aks command invoke --subscription <subscription> -g <aks-resource-group> -n <aks-cluster> --command \
"kubectl get deploy -A -o custom-columns=NS:.metadata.namespace,IMAGE:.spec.template.spec.containers[0].image --no-headers | grep memex-portal-ai"
.github/scripts/check-image-set.sh <short-sha> # the exact assertion CD itself makes
🚨 …and the RIGHT IMAGE is still not the right ANSWER — the grain serves what it activated with
"The grain keeps serving old state until we send a dispose request" (maintainer, 2026-09-17).
A per-node hub binds its configuration ONCE while activating and is then pinned by address — routing
never resolves that path again. A Roll or a Restart does end the activations on the pods it
replaces, so those addresses re-read; the gap is everything a portal absorbs while it keeps
running (a package installed or updated, a NodeType recompiled in place, a sync landing new
source), and what the fresh activation then chooses. The rolled tag is necessary and not
sufficient.
Concretely, after a roll or a package publish:
- 🚨 Read the EFFECTIVE
Modules:AutoRecycleOnStaleBuild— the code default and the fleet disagree. The runtime default is off, where the stale-build watcher publishes only a banner offering a recycle and a portal stays a mixture of old and new assemblies for as long as viewers do not click (the 2026-08-25 Store outage is what that costs). Butdeploy/aks/values.aks.yamlsets it true fleet-wide, so the portals this repo deploys converge by themselves. On neither setting does it fire for a same-path byte change. - Verify by EXERCISING the feature at the running address, not by the tag, the install record or
the green tick. Then, if it still answers the old way, recycle that address — the
recycleverb (MCP tool, the node's Recycle menu entry,mw recycle <path>) orhub.RecycleNode(path, reason: "…")in code — and exercise it again. - The two surfaces differ.
hub.RecycleNodeis dispose-only. The operatorrecycleverb also stampsRequestedReleaseAt+RequestedReleaseForce, but only when the target is a NodeType node — and a FORCED release is precisely the one that skips on-demand prebuilt adoption and compiles the live source (#2818), so it is the operator's rebuild-from-source remedy. Recycling an ordinary page recompiles nothing. - 🚨 A recycle makes the activation RE-READ; it does not decide what the re-read FINDS. A third
recycle finds nothing a second did not: a dispose-only recycle re-resolves the same store key
(nodeTypePath, version)and re-binds the same local copy (#2471 — measured over 30+ minutes and six recycles), and an UNFORCED trigger, or aModules:RequirePrebuiltmesh, re-adopts the same bundle — remedy there is a rebake and republish for THIS framework identity. And no recycle fixes a package that never reached the pod: read the[ModuleLoad] <Assembly> ← … (written=…)line before recycling anything.
Full reference: StaleStateUntilRecycle.md.
Running the portal locally (not a deploy route)
dotnet watch --project ../MeshWeaver.Plugins/src/Memex.AppHost restarts only the affected Aspire
resource on save; the dashboard's Resources → ⋯ → Restart is the fallback; a process kill is the
last resort. Don't kill the whole aspire / Memex.AppHost process unless you changed AppHost
wiring itself — a full restart costs 30–60 s and loses the dashboard auth token. Full reference:
LocalDevWorkflow.md.
Checklist
- Route chosen by target (AKS vs Container Apps) — no
tools/deploy.sh/aspire deployagainst AKS. - The operation was filed as a
Hosting/InstanceActionon the control instance (Reconcile/Roll, Sample, Logs, Audit); anykubectlyou ran yourself is written up as break-glass, with its other half reconciled. - Where
kubectlwas unavoidable, it was reached only throughaz aks command invoke, with--subscription <subscription>. - A gated action was approved on the PLAN it parked with (the plan hash on the page), not on its reason.
- No
deploy.shre-run for a code update. - The migration Job's log shows
Database migration completed. Version: N. - The RUNNING image tag was read back off the deployment — not inferred from a green CI tick.
- The feature was EXERCISED at the running address, and any address still serving the old state was recycled (and named in the write-up) — a rolled image is not a re-read activation.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 466k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 466k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 466k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |