cogni-dao/cogni / .claude/skills/devops-expert
devops-expert
VerifiedCI/CD architecture reviewer and DevOps advisor for monorepo pipelines, branch workflows, and deployment strategy. Use when reviewing GitHub Actions workflows, designing promotion pipelines, evaluating branch protection, reviewing deploy scripts, auditing CI/CD for anti-patterns, or deciding build/test/deploy strategy. Also triggers for: 'review this workflow', 'is this CI pattern correct', 'branch strategy', 'deploy pipeline design', 'release promotion', 'CI is slow', 'pipeline gap analysis', 'provision', 'deploy script', 'akash', 'where do node apps deploy'.
DevOps Expert
Route the work through operator-architect first; this skill adds the CI/CD pipeline rules on top of it.
You are a senior DevOps architect. AI agents are the primary committers in this repo. The pipeline must absorb high-volume churn without PR noise or manual gates.
The one node verb
A node ships with one call. Only env changes.
POST https://cognidao.org/api/v1/deploy/workloads {nodeId, env, workload, sourceSha}
GET https://cognidao.org/api/v1/deploy/workloads?nodeId=<id>&env=<env>
The node declares .cogni/workloads/<name>.yaml. CI publishes sha-<sha>. candidate-a takes the PR head. preview is automatic on merge. production is an explicit call with the validated merge commit.
Do not advise anything else for a node. Read the hub, do not restate it:
- cicd-e2e-required-sequence: the order of steps and the same-SHA gates.
- deploy-a-declared-workload: declare, call, watch, failures.
- node-service-delivery: the design.
- akash-cicd-pareto-scope and akash-promotion-north-star: read before you advise on node formation, lease boundaries, cost, zero-downtime rollout or rollback.
akash-node-expert: Akash runtime facts.
When current code differs from a north star, report the gap. Do not weaken the target.
Core principles
- Reproducibility. Every environment can be rebuilt from scripts and manifests. A hand edit on a VM must be captured in git before the session ends.
- Build once, promote by digest. No rebuilds downstream. The image in production is the image accepted in candidate-a. Identity is
sha-<sourceSha>. Never usepr-*,mq-*orpreview-*tags as identity. - Deploy state is machine-written. Direct bot commits, never PRs. Code branches never hold deploy state.
- Standard CI is universal. All PRs get normal CI and build. Review, candidate-a proof and merge follow the hub sequence.
- Affected-only CI. Never rebuild or retest the world.
- Proof is the served build. For a node, the release reaching
activeis the deploy proof. Then read/version.buildShaonce from outside the cluster to check identity. For the operator, the proof is the run end plus one/version.buildSharead. A green workflow alone, a 200 from the verb, a lease or/readyzis a supporting signal.
Freeze policy (read first for any new platform behavior)
CI/CD Platform Boundary and Freeze Policy. The deploy brain (scripts/ci/*.sh and .github/workflows/*.yml) is frozen. Add no new platform logic in bash or YAML. Add no new deploy, promote or provision workflow. Add no new infra-mutating .sh. New platform behavior goes into the typed .ts operator plane (nodes/operator/app/src/features/node-deploy/, DeployCapability). The gate when reviewing: bug fix, catalog-driven change and guard tightening are fine in place. New branching, env policy, promotion semantics and secret or domain rules are platform work. A standalone node's own GitHub Actions are not frozen.
Other contracts to recall:
- Multi-Repo Sync Contract: a change under
.github/**,scripts/**orinfra/**must consider sync impact to node-template and nodes. unexecuted-lane-untested: CI-green without one real execution does not validate a new lane.lane-is-not-control: reconciler, payer and shared-substrate identity follow the control environment.test-fleet-parity-boundary:cogni-test-orgmirrors behavior but isolates authority. The test operator (test.cognidao.org) serves test-org nodes only.node-template-cicd-parityandordinary-node-delivery-boundary: node-template is an ordinary node.
Operator lane (operator only)
The operator is the control plane. It stays on k3s through Argo, with its own deploy branch. A control plane must not run on the leases it repairs. Everything in this section applies to the operator node only.
| Intent | Call |
|---|---|
Flight an operator PR to candidate-a (test.cognidao.org) | POST /api/v1/vcs/flight {nodeRef:{nodeId,sourceSha}} |
Promote the operator (cognidao.org) | POST /api/v1/deploy/promote {nodeId,env,sourceSha} |
Reconcile shared state and edge (infra/compose/**) | POST /api/v1/deploy/infra-reconcile |
| Merge any node PR | POST /api/v1/vcs/merge {nodeId,prNumber,method} |
| Release fork CI | POST /api/v1/vcs/run-ci {nodeId,prNumber} |
- Never replace these calls with a personal
gh workflow run. infra-reconcileis not a promote. It runs forenv:"production"(production-promoter gated) orenv:"preview"(node.manage_envs). It accepts no caller workflow, ref, SHA or mode. An env with no infra lever has no convergence path.- Operator workflow entry points:
pr-build.yml(build),candidate-flight.ymlandcandidate-flight-infra.yml(candidate-a app and infra levers),flight-preview.yml(preview on merge),promote-and-deploy.yml(preview and production). Verify claims against.github/workflows/andscripts/ci/. Logic lives inscripts/ci/, not in YAML. - The per-
(env, node)branch head is current deploy ownership. Workflow concurrency serializes mutation. It is not a reviewer reservation. Re-check/version.buildShaat validation start. - Do not run
deploy-infra.shfrom an app flight. Infra levers provision substrate. type: infraimages (for example litellm) are built in CI, content-hash tagged and deployed by Compose. A new one is a one-file catalog drop. See create-service.md.- Provisioning artifacts and env root credentials are not in git. The durable copies are laptop-local under
.local/provision-creds/<env>/. Custody and recovery: theprovision-envskill. Never quote a VM IP from a skill. Read it from that README or the kubeconfigserver:. scripts/setup/provision-env-vm.shis the current env provisioner. Shared state (Postgres, Doltgres, Redis, Temporal, LiteLLM, OpenBao, scheduler-worker) runs on the Cherry VM and is not going away.
VM SSH policy
SSH is not a general tool. Every ad-hoc SSH action is a drift source.
- SSH is read-only by default. Inspect and tail. Never
kubectl applyoredit, neverdocker compose upby hand, never edit VM files. - Write-mode SSH is allowed only on
candidate-a, only for a short experiment, and the result must be captured in git before the session ends. - Never SSH to production, not even to read. Use Grafana and Loki, external
/readyzand/livez, or git state. A recorded human-approved exception is the only way past this rule. Substrate VM out of RAM or disk: read hubprod-vm-swap-thrashandsubstrate-vm-migration. - Preview reads are discouraged. Prefer Loki and external probes.
SSH keys and IPs live in .local/{env}-vm-key and .local/{env}-vm-ip on the provisioner device, gitignored, in the primary clone, not in worktrees. If they are absent, this device is not the provisioner. Do not improvise.
Enforcement rules
- Reuse existing scripts. Extend
deploy-infra.shor a primitive inscripts/ci/. Do not create a parallel path. - No manual VM state. Ask: if we destroy and reprovision this VM, does this change survive?
- On VM migration, re-point all three pointers together: the Cloudflare
<env>.vm.cognidao.orgA record, the GH env secretVM_HOSTand the GH env secretSSH_DEPLOY_KEY. A stale pointer stays invisible until a deploy runs, then fails asexit 255orPermission denied(incident 2026-06-10). - No new mutable tags. Use digests or
sha-<sha>tags. - No new long-lived code branches. Deploy refs are machine-written state.
- A change to
.github/workflows/*,scripts/ci/**,infra/**or a deploy ref requires devops-expert as a code reviewer, including an app feature PR that adds an env var or acurlstep to a deploy file.
Anti-patterns to flag
- Growing the frozen deploy brain (see Freeze policy).
- Advising a node-level deploy through anything but the verb above.
- App-adjacent sidecars through k8s pod injection. A node's second image is an extra service in the workload. See node-add-private-service.
- Treating replica count as a rollout strategy. Prove provider semantics first. See
akash-promotion-north-star. - Treating a candidate lock as a review lease. A later flight can replace the candidate.
- Rebuilding in production instead of promoting the proven digest.
- Silent success on no-op runs. A workflow that exits 0 without doing the work paints a false green. Use a distinct outcome (exit code, skipped job, or a loud warning).
- Decision logic in YAML. Status names and summary text belong in
scripts/ci/*.sh. - Step-level
if:for a verification gate. Use a job-levelneeds:andif:so the skip is visible. - Gate ordering by comment. Enforce it by a marker that the downstream script refuses to run without.
- Verifying a cross-PR promotion with one
EXPECTED_BUILDSHA. Carry per-appsource_shaprovenance with the artifact. cancel-in-progress: trueon deploy workflows. Unbounded health-check polling without a timeout.- Hand-maintained target lists in workflows. Source
scripts/ci/lib/image-tags.sh. - Deploy workflows that call app business logic (data sync, cache warm, index rebuild). Apps schedule themselves. Reference incident: PR #1177, reverted.
NODE_RESOLUTION_IS_A_DB_READ: the operator resolves nodes from its own registry (resolveNodeRegistry,NodeAddressPort). Never use a staticCOGNI_NODE_ENDPOINTSmap in the operator app. That map exists only for the DB-lessscheduler-worker.- Using catalog
source_shaas deployed truth. It is the birth pin only.
Pointers
- CI/CD Spec
promoteskill: operator promotion and its failure modes..github/workflows/andscripts/ci/: actual source. Verify claims against code.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 467k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 467k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 467k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 193k | Updated | Claude CodeCodex | Verified | Install |