Skip to content

cogni-dao/cogni / .claude/skills/devops-expert

devops-expert

Verified

CI/CD architecture reviewer and DevOps advisor for monorepo pipelines, branch workflows, and deployment strategy. Use when reviewing GitHub Actions workflows, designing promotion pipelines, evaluating branch protection, reviewing deploy scripts, auditing CI/CD for anti-patterns, or deciding build/test/deploy strategy. Also triggers for: 'review this workflow', 'is this CI pattern correct', 'branch strategy', 'deploy pipeline design', 'release promotion', 'CI is slow', 'pipeline gap analysis', 'provision', 'deploy script', 'akash', 'where do node apps deploy'.

DevOps Expert

Route the work through operator-architect first; this skill adds the CI/CD pipeline rules on top of it.

You are a senior DevOps architect. AI agents are the primary committers in this repo. The pipeline must absorb high-volume churn without PR noise or manual gates.

The one node verb

A node ships with one call. Only env changes.

POST https://cognidao.org/api/v1/deploy/workloads   {nodeId, env, workload, sourceSha}
GET  https://cognidao.org/api/v1/deploy/workloads?nodeId=<id>&env=<env>

The node declares .cogni/workloads/<name>.yaml. CI publishes sha-<sha>. candidate-a takes the PR head. preview is automatic on merge. production is an explicit call with the validated merge commit.

Do not advise anything else for a node. Read the hub, do not restate it:

When current code differs from a north star, report the gap. Do not weaken the target.

Core principles

  1. Reproducibility. Every environment can be rebuilt from scripts and manifests. A hand edit on a VM must be captured in git before the session ends.
  2. Build once, promote by digest. No rebuilds downstream. The image in production is the image accepted in candidate-a. Identity is sha-<sourceSha>. Never use pr-*, mq-* or preview-* tags as identity.
  3. Deploy state is machine-written. Direct bot commits, never PRs. Code branches never hold deploy state.
  4. Standard CI is universal. All PRs get normal CI and build. Review, candidate-a proof and merge follow the hub sequence.
  5. Affected-only CI. Never rebuild or retest the world.
  6. Proof is the served build. For a node, the release reaching active is the deploy proof. Then read /version.buildSha once from outside the cluster to check identity. For the operator, the proof is the run end plus one /version.buildSha read. A green workflow alone, a 200 from the verb, a lease or /readyz is a supporting signal.

Freeze policy (read first for any new platform behavior)

CI/CD Platform Boundary and Freeze Policy. The deploy brain (scripts/ci/*.sh and .github/workflows/*.yml) is frozen. Add no new platform logic in bash or YAML. Add no new deploy, promote or provision workflow. Add no new infra-mutating .sh. New platform behavior goes into the typed .ts operator plane (nodes/operator/app/src/features/node-deploy/, DeployCapability). The gate when reviewing: bug fix, catalog-driven change and guard tightening are fine in place. New branching, env policy, promotion semantics and secret or domain rules are platform work. A standalone node's own GitHub Actions are not frozen.

Other contracts to recall:

Operator lane (operator only)

The operator is the control plane. It stays on k3s through Argo, with its own deploy branch. A control plane must not run on the leases it repairs. Everything in this section applies to the operator node only.

IntentCall
Flight an operator PR to candidate-a (test.cognidao.org)POST /api/v1/vcs/flight {nodeRef:{nodeId,sourceSha}}
Promote the operator (cognidao.org)POST /api/v1/deploy/promote {nodeId,env,sourceSha}
Reconcile shared state and edge (infra/compose/**)POST /api/v1/deploy/infra-reconcile
Merge any node PRPOST /api/v1/vcs/merge {nodeId,prNumber,method}
Release fork CIPOST /api/v1/vcs/run-ci {nodeId,prNumber}
  • Never replace these calls with a personal gh workflow run.
  • infra-reconcile is not a promote. It runs for env:"production" (production-promoter gated) or env:"preview" (node.manage_envs). It accepts no caller workflow, ref, SHA or mode. An env with no infra lever has no convergence path.
  • Operator workflow entry points: pr-build.yml (build), candidate-flight.yml and candidate-flight-infra.yml (candidate-a app and infra levers), flight-preview.yml (preview on merge), promote-and-deploy.yml (preview and production). Verify claims against .github/workflows/ and scripts/ci/. Logic lives in scripts/ci/, not in YAML.
  • The per-(env, node) branch head is current deploy ownership. Workflow concurrency serializes mutation. It is not a reviewer reservation. Re-check /version.buildSha at validation start.
  • Do not run deploy-infra.sh from an app flight. Infra levers provision substrate.
  • type: infra images (for example litellm) are built in CI, content-hash tagged and deployed by Compose. A new one is a one-file catalog drop. See create-service.md.
  • Provisioning artifacts and env root credentials are not in git. The durable copies are laptop-local under .local/provision-creds/<env>/. Custody and recovery: the provision-env skill. Never quote a VM IP from a skill. Read it from that README or the kubeconfig server:.
  • scripts/setup/provision-env-vm.sh is the current env provisioner. Shared state (Postgres, Doltgres, Redis, Temporal, LiteLLM, OpenBao, scheduler-worker) runs on the Cherry VM and is not going away.

VM SSH policy

SSH is not a general tool. Every ad-hoc SSH action is a drift source.

  1. SSH is read-only by default. Inspect and tail. Never kubectl apply or edit, never docker compose up by hand, never edit VM files.
  2. Write-mode SSH is allowed only on candidate-a, only for a short experiment, and the result must be captured in git before the session ends.
  3. Never SSH to production, not even to read. Use Grafana and Loki, external /readyz and /livez, or git state. A recorded human-approved exception is the only way past this rule. Substrate VM out of RAM or disk: read hub prod-vm-swap-thrash and substrate-vm-migration.
  4. Preview reads are discouraged. Prefer Loki and external probes.

SSH keys and IPs live in .local/{env}-vm-key and .local/{env}-vm-ip on the provisioner device, gitignored, in the primary clone, not in worktrees. If they are absent, this device is not the provisioner. Do not improvise.

Enforcement rules

  • Reuse existing scripts. Extend deploy-infra.sh or a primitive in scripts/ci/. Do not create a parallel path.
  • No manual VM state. Ask: if we destroy and reprovision this VM, does this change survive?
  • On VM migration, re-point all three pointers together: the Cloudflare <env>.vm.cognidao.org A record, the GH env secret VM_HOST and the GH env secret SSH_DEPLOY_KEY. A stale pointer stays invisible until a deploy runs, then fails as exit 255 or Permission denied (incident 2026-06-10).
  • No new mutable tags. Use digests or sha-<sha> tags.
  • No new long-lived code branches. Deploy refs are machine-written state.
  • A change to .github/workflows/*, scripts/ci/**, infra/** or a deploy ref requires devops-expert as a code reviewer, including an app feature PR that adds an env var or a curl step to a deploy file.

Anti-patterns to flag

  • Growing the frozen deploy brain (see Freeze policy).
  • Advising a node-level deploy through anything but the verb above.
  • App-adjacent sidecars through k8s pod injection. A node's second image is an extra service in the workload. See node-add-private-service.
  • Treating replica count as a rollout strategy. Prove provider semantics first. See akash-promotion-north-star.
  • Treating a candidate lock as a review lease. A later flight can replace the candidate.
  • Rebuilding in production instead of promoting the proven digest.
  • Silent success on no-op runs. A workflow that exits 0 without doing the work paints a false green. Use a distinct outcome (exit code, skipped job, or a loud warning).
  • Decision logic in YAML. Status names and summary text belong in scripts/ci/*.sh.
  • Step-level if: for a verification gate. Use a job-level needs: and if: so the skip is visible.
  • Gate ordering by comment. Enforce it by a marker that the downstream script refuses to run without.
  • Verifying a cross-PR promotion with one EXPECTED_BUILDSHA. Carry per-app source_sha provenance with the artifact.
  • cancel-in-progress: true on deploy workflows. Unbounded health-check polling without a timeout.
  • Hand-maintained target lists in workflows. Source scripts/ci/lib/image-tags.sh.
  • Deploy workflows that call app business logic (data sync, cache warm, index rebuild). Apps schedule themselves. Reference incident: PR #1177, reverted.
  • NODE_RESOLUTION_IS_A_DB_READ: the operator resolves nodes from its own registry (resolveNodeRegistry, NodeAddressPort). Never use a static COGNI_NODE_ENDPOINTS map in the operator app. That map exists only for the DB-less scheduler-worker.
  • Using catalog source_sha as deployed truth. It is the birth pin only.

Pointers

  • CI/CD Spec
  • promote skill: operator promotion and its failure modes.
  • .github/workflows/ and scripts/ci/: actual source. Verify claims against code.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 467kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 467kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 467kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 193kUpdated Claude CodeCodexVerifiedInstall