cogni-dao/cogni / .claude/skills/akash-node-expert
akash-node-expert
VerifiedAkash runtime canon for how Cogni nodes run on decentralized compute. Use when working on Akash placement, leases, providers, bids, escrow, compute egress, or debugging a node that will not come up on Akash. Nodes ship through POST /api/v1/deploy/workloads; the hub owns that journey. Triggers on: 'akash', 'lease', 'placement', 'bid', 'provider', 'escrow', 'zencloud', 'egress', 'node-deploy', 'sequencer'.
Akash Node Expert
Route the work through operator-architect first; this skill adds the Akash runtime rules on top of it.
How Cogni nodes run on decentralized compute. This skill holds Akash runtime facts. The hub holds the delivery journey.
How a node ships
A node declares .cogni/workloads/<name>.yaml and calls POST /api/v1/deploy/workloads {nodeId, env, workload, sourceSha}. Only env changes. The operator sequencer does create, prove, edge flip, retain and close. Read before acting:
- cicd-e2e-required-sequence for the order and gates.
- deploy-a-declared-workload for the declaration, the watch and the failure signatures.
- node-service-delivery for the design.
Code: nodes/operator/app/src/features/node-deploy/ (read its AGENTS.md). The operator itself stays on k3s and Argo (operator only). Never roll a node back to k3s. Fix forward.
Do not trust any roster written in a skill. Read live state: GET /api/v1/nodes, infra/catalog/*.yaml, curl https://<host>/version.
Scaling north star
Before you design placement, lease lifecycle, capacity, cost or shared substrate, read akash-cicd-pareto-scope (story.5024). Its binding decisions:
- Spawn ends with the production hostname serving the exact immutable SHA. Candidate-a is a passive proof gate. Preview is absent at birth.
- One Akash deployment per
(node, environment), not one per service and not one giant fleet lease. - Pool generic substrate. The controller's recovery path stays outside the leases it reconciles.
When as-built code differs, name the gap and fix toward the target.
Read ground truth first
A 2026-10-01 session produced five wrong root causes for one node by inferring topology from logs. Each died to a read that was available the whole time.
- Release state.
GET /api/v1/deploy/workloads?nodeId=<id>&env=<env>shows the desired release and each attempt. The provider and lease id are in the attempt. - Sequencer log.
{env="<env>",service="sequencer"} |= "node_deploy_step" |= "<releaseId>". A good release runsattempt_plannedtoblue_retiring. - Money truth.
scripts/ops/recover-orphaned-akash-lease.sh --auditreads Console with the actuator credential. The--auditsummary under-reports. The per-DSEQ read (DRY_RUN=1 --dseq <dseq>) is the truth (bug.5262).{env="<env>"} |= "compute_cost_observed"in Loki proves the money stopped. - Database truth.
scripts/grafana-postgres-query.shruns read-only SQL with no SSH.pg_stat_activityis server-wide. Alwaysgroup by datname. A node datasource may fail auth (bug.5117, SQLSTATE 28P01). Use--node operatorwith adatnamefilter.
Placement narrows a pool; it is not egress
required_placement_countriesfilters bids on the provider's declared ingress country. That is not egress. Measure egress from inside a lease when a third party geo-fences by IP.- Placement lives in the workload's placement spec, not the SDL. Changing the country set does not change the SDL.
featEndpointCustomDomainis optional. Many audited providers publishfalseand never serve a custom host. Both Swiss providers declarefalse.activeLeases > 0is not a quality signal (bug.5334). It made a provider's first lease unwinnable.ipCountryCodeis GeoIP and it lies. Screen on the declared field.- Read the roster before you widen any gate.
NO_ELIGIBLE_BIDScarries[refused: <reason>=N]and[bids: ...]. First-match-wins attribution hides later gates. - Market scale: about 1850 registered, about 60 online, about 33 audited, about 28 custom-domain capable across 9 countries. Pass a country set, never one country.
Probe the market with the dev key
.env.cogni carries AKASH_PROBE_CONSOLE_API_KEY and AKASH_PROBE_ACCOUNT_ID, a dedicated dev wallet. It answers "would X bid" in 2 minutes. Plain HTTP on https://console-api.akash.network with x-api-key:
POST /v1/deployments {data:{sdl, deposit:0.5}}returns{dseq, manifest}GET /v1/bids?dseq=<dseq>POST /v1/leases {manifest, leases:[...]}GET /v1/deployments/<dseq>showsleases[].status.services.<svc>.urisDELETE /v1/deployments/<dseq>closes
Never use the production actuator account. A manual mint has no akash_tx_allocations receipt. It cannot be closed through the actuator and it bills invisibly (bug.5189, bug.5262). Always DELETE, then re-read state until closed. A 200 from DELETE is not proof. Price is never the blocker: a bare SDL draws 9 to 11 bids in 10 to 20 seconds.
Facts that stay true
- The
akash-tx-actuatoris the only Akash wallet writer. One wallet, one active writer, enforced by a Postgres single-writer index. - A lease outlives every object that names it. Never strip a finalizer to "unwedge" something before you close the lease, verify by re-reading Console, and only then clear. Use
scripts/ops/recover-orphaned-akash-lease.sh. It refuses to clear what it cannot prove closed. - The cluster is not the source of truth for spend. Audit the wallet.
- A terminally closed lease's idempotency key is spent. The verb uses a new
releaseIdand attempt for each retry. - A
stateworkload keeps one long-lived lease and is never auto-replaced. App and worker workloads recover on a chain-confirmed close, oninsufficient_funds, or after 15 minutes of failed proof. - An empty node database shows as a generic
INTERNAL_ERRORon/readyz. The proof fails and the sequencer retries. When one env works and another does not, diff the envs first. - Lease logs reach Loki through the lease-log pump (bug.5240) as
{env, node=<uuid>, workload=<manifest name>, service=<sdl-service = Score container key>, source="lease"}. Read them withGET /api/v1/nodes/{id}/observability/logs?service=<name>[&workload=<name>]. On an env with no pump, Loki silence proves nothing. - Fork CI checks out the PR head, not a merge commit. Files that landed on main mid-PR must be copied onto the PR branch (bug.5124).
Poly production placement findings, 2026-11-02
- Gen-24 and gen-25 failed
provider_rejectedwith zero FI, NL or PT bids, although the same spec drew FI bids from the dev wallet. Gen-26 leased RHITE FI. The cause is unknown. Read the provider roster first, then probe with the dev key. - Providers froggy (RO) and metz (CH) returned nginx 404 for every host. Do not keep a static list of them. PR #3056 set the provider allowlist (
AKASH_ALLOWED_PROVIDERS) to empty. Instead, a new app create skips each provider with one confirmedhost_not_routedstrike on any lane in the last 24 hours (PR #3061, bug.5586). The no-lockout rules innodes/operator/app/src/features/node-deploy/AGENTS.md(Provider memory) still apply. - The Polymarket CLOB geoblock covers only Crimea, Donetsk and Luhansk in UA. UA providers are usable.
Operator only: control and state substrate
The operator, the scheduler-worker and the Compose substrate (Postgres, Doltgres, Temporal, LiteLLM, OpenBao, OpenFGA, Caddy) run on Cherry. The permanent rule is that the controller and its recovery path cannot self-host only on leases they manage. Move stateless shared services first. Move stateful services only after proven backup, restore and cross-provider recovery.
Open edges
Check work items before you assume. A second audited provider is needed (task.5075). Every lane reaches the shared substrate over its env mesh; the VM firewall admits no provider egress IP (task.5428). Database migrations before a release serves: task.5408. Stop a workload with DELETE /api/v1/deploy/workloads (task.5491, hub deploy-a-declared-workload section 7); retained state also needs destroyRetainedData and the live dseq. It is live only after task.5491 is promoted; until then no stop verb exists. A stop leaves the host record in DNS (bug.5633).
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |