jackson2w/claude-code-skills / archive/openclaw-deployment
openclaw-deployment
UnverifiedThis skill should be used when deploying or debugging a self-hosted OpenClaw (formerly Clawdbot/Moltbot) agent gateway — a Node.js personal-assistant service bridging Telegram/WhatsApp/Discord to an LLM with tool/skill access. Covers install, secure baseline config (loopback bind, token auth, exec approval gating), the two non-obvious config traps that silently break things (model selection vs. API key, exec security vs. ask), Telegram channel lockdown, configuring the web_search tool/Brave provider, voice-note transcription (tools.media.audio/Groq), memory indexing and its embedding-provider auth (openclaw memory index/status), evaluating ClawHub skills/plugins by real usage data, and building trustworthy custom skills instead of pulling from the unvetted ClawHub marketplace. Trigger phrases include "openclaw config", "openclaw gateway", "tools.exec.security", "openclaw model not found", "ProviderAuthError No API key found for provider openai", "exec denied security=deny", "openclaw dashboard", "openclaw pairing", "ClawHub skill", "openclaw skills install local", "openclaw web search", "tools.web.search.provider", "web_search provider is not available", "openclaw plugins install clawhub", "plugins.allow is empty", "tools.media.audio", "voice memo transcription", "openclaw skills search --json", "ClawHub popular skills", "EXTERNALLY-MANAGED pip openclaw", "AgentMail openclaw", "clawdbot paths in skill", "openclaw memory index", "memorySearch.provider", "memory index failed no api key", "credit_balance_exhausted", "insufficient_quota embeddings", "memory index --force", "scoped sudo for openclaw agent", "dispatch script security boundary", "openclaw agent --message-file", "openclaw agent --session-key timeout", "brief openclaw on new capabilities", "openclaw treats primer as prompt injection", "openclaw session jsonl toolCall", "arm olu with new tools", "expand openclaw agent privilege incrementally", "skill_workshop", "openclaw agent rejects everything as injected",
OpenClaw deployment
Built and verified end-to-end 2026-08-16 deploying a personal instance on a Vultr VPS
(dfw), wired to Telegram, Anthropic, GitHub, and Cloudflare. Every gotcha below was hit for
real — found by reading actual error logs and openclaw config schema, not by trusting docs
pages (several official doc pages returned "not found in this excerpt" for exactly the config
keys that mattered most; the installed CLI's own config schema output is the real source of
truth).
Install
# Node.js via NodeSource (nodistro repo works fine on Debian 13/trixie, see
# proxmox-node-systemd-service skill for the full apt-repo setup sequence)
npm install -g openclaw@latest
openclaw --version
Run as a dedicated non-root, non-login system user (adduser --disabled-password --disabled-login openclaw; passwd -l openclaw), managed via a systemd unit with User=openclaw,
ProtectSystem=strict, PrivateTmp=true, ReadWritePaths=/home/openclaw/.openclaw.
ReadWritePaths= is space-separated and grows as OpenClaw needs write access elsewhere — e.g.
the homelab's /srv/agent-exchange/to-claude (a filesystem-based cross-agent handoff channel,
one-way by design: only that specific subpath is writable, not its sibling to-olu) was added
alongside the .openclaw path on 2026-08-23. Verify any such addition against real confinement
(systemd-run matching the unit's properties + a check of the live process's
/proc/<pid>/mountinfo), not a sudo -u openclaw shortcut — see "NoNewPrivileges=yes
silently kills sudo" below for why that shortcut misses confinement bugs generally. Full
detail on the agent-exchange channel specifically is in the homelab planning repo's
project_agent_exchange_channel memory, not this skill.
Every path in ReadWritePaths= must already exist on disk before the unit starts —
systemd builds the mount namespace before ExecStart runs, and a missing path fails the whole
start with Failed to set up mount namespacing: <path>: No such file or directory /
Failed at step NAMESPACE, then crash-loops on Restart=always. It does not create the
directory for you, unlike ReadWritePaths= in some other tools' documentation examples.
Confirmed 2026-08-31 wiring OpenClaw through Agent Vault's agent-vault run -- wrapper (see
the agent-vault-credential-broker skill): added /home/openclaw/.agent-vault to
ReadWritePaths= for the wrapper's CA-cert cache, but the directory didn't exist yet — instant
crash loop, caught within a couple restart cycles and rolled back. Create any new
ReadWritePaths= target first (mkdir -p, correct ownership) before adding it to the unit,
never as a "the process will create it" assumption.
Gotcha 1 — setting the API key does NOT select which model gets used
Configuring models.providers.anthropic.apiKey is necessary but not sufficient. Without also
setting a default model, the gateway silently defaults to openai/gpt-5.5 — and if no OpenAI key
is configured (because you're using Anthropic), every request fails:
ProviderAuthError: No API key found for provider "openai". Auth store: ...
model fallback decision: decision=candidate_failed requested=openai/gpt-5.5 ... reason=auth
This looks like an Anthropic auth problem. It isn't — the Anthropic key is fine, the model selection just defaulted somewhere else entirely. Fix:
{ agents: { defaults: { model: "anthropic/claude-sonnet-5" } } }
Don't guess the model string — OpenClaw maintains its own alias registry that doesn't necessarily match Anthropic's raw API model IDs, and it changes over time. Get the current valid list straight from the installed CLI:
openclaw models list
Gotcha 2 — tools.exec.security: "deny" is a HARD BLOCK, not an approval gate
The two exec-control fields look like they compose, but they don't the way you'd expect:
tools.exec.security:deny|allowlist|full— eligibility gate.denyrefuses every exec call outright, unconditionally, beforeaskis ever consulted.tools.exec.ask:off|on-miss|always— approval-prompt control. Only matters for calls that already passed thesecuritygate.
If you want "the agent can run anything, but must ask permission every time" (a common and sensible posture for a personal assistant with real infra access), the correct pairing is:
{ tools: { exec: { security: "full", ask: "always" } } }
Not security: "deny" — that sounds like the safer/stricter choice and is the literal value
shown in some published "secure baseline" examples, but it silently refuses everything with no
prompt ever sent. Symptom in the logs when this is misconfigured:
[tools] exec failed: exec denied: host=gateway security=deny raw_params={...}
[tools] exec failed: exec denied: host=sandbox security=deny raw_params={...}
Both host=gateway and host=sandbox get tried and both refuse — no approval request is ever
generated. Confirm the fix actually worked by watching for the real approval flow in the logs
after a live test:
[ws] ⇄ res ✓ exec.approval.request 68ms ...
[ws] ⇄ res ✓ exec.approval.waitDecision 8946ms ...
The waitDecision timing (matches how long the human actually took to respond) is the tell that
it's genuinely gating on a live decision, not rubber-stamping.
When in doubt about any enum's valid values, don't trust doc pages piecemeal — pull the schema directly from the installed binary:
openclaw config schema > /tmp/openclaw-schema.json
python3 -c "
import json
s = json.load(open('/tmp/openclaw-schema.json'))
def find(o, path=''):
if isinstance(o, dict):
if 'enum' in o: print(path, o['enum'])
for k,v in o.items(): find(v, path+'/'+k)
elif isinstance(o, list):
for v in o: find(v, path)
find(s)
"
Gotcha 3 — web_search providers aren't bundled; ClawHub install is required even for "official" ones
Unlike the CLI-wrapper skills (github, weather), which ship bundled and just need a binary
(gh) present, tools.web.search providers like Brave are not in the base install.
openclaw plugins list won't show brave at all until it's installed — the plugin registry it
prints only covers model providers + chat channels, not search providers. Setting
tools.web.search.provider: "brave" (or running openclaw config validate/config set) before
installing fails with a misleading error:
tools.web.search.provider: web_search provider is not available: brave
(install or enable plugin "brave", then run openclaw doctor --fix)
openclaw plugins enable brave also fails at this point (Plugin not found: brave) — enable
only toggles a plugin that's already discovered on disk, it doesn't install one. The actual fix
is openclaw plugins search <provider> to find the ClawHub package (prefer the official-tagged
@openclaw/<name>-plugin entry over community alternatives, same trust reasoning as the ClawHub
section below), then:
openclaw plugins install clawhub:@openclaw/brave-plugin
openclaw config set tools.web.search.provider brave
sudo systemctl restart openclaw.service # plugin install requires a gateway restart to load
openclaw doctor --fix does not fix this — when it hits the same "provider not available"
error, it silently deletes the offending tools.web.search block from config instead of
installing/enabling the plugin, and reports success. Always diff the config before/after running
--fix; don't assume it did the intelligent thing.
A new ClawHub plugin install also triggers a standing warning once plugins.allow is empty
(the default): discovered non-bundled plugins may auto-load: brave ... To trust them explicitly, set plugins.allow. Fix: read the exact active set from the [gateway] http server listening (N plugins: ...) log line after a restart — openclaw plugins list alone overstates it, since it
shows every discovered bundled plugin (51 on a stock install: unused providers like
azure-speech, cohere, deepgram, etc. all show enabled), not just the ones actually loaded
at runtime. Set plugins.allow to the smaller runtime-active list (confirm it's stable across at
least two independent restarts before trusting it), e.g.:
openclaw config set plugins.allow '["brave","browser","canvas","device-pair","file-transfer","memory-core","ollama","phone-control","talk-voice","telegram"]' --json
The schema notes bundled chat channel plugins (telegram, etc.) auto-activate when their channel
is explicitly enabled even if left off plugins.allow — but that exception doesn't cover
non-channel bundled plugins (browser, canvas, memory-core, ...), so list those explicitly
too; a partial list silently breaks whatever's left off. This kind of config set mid-session can
land while a real conversation is in flight (Telegram) — the gateway's reload logic detects that
and defers the restart until the in-flight turn/background task finishes
([reload] restart blocked by active background task run(s)) rather than dropping it, so it's
safe to apply without warning the user first. After restart, diff the plugin list in the log
line against the pre-change baseline (should be identical), then re-run the same
openclaw agent --json / toolSummary live check to prove the allowlist didn't silently break
what you were trying to protect.
Ad-hoc sudo -u openclaw openclaw ... CLI commands don't see the service's env vars.
Provider API keys (BRAVE_API_KEY, etc.) typically live in a file wired into the systemd unit via
EnvironmentFile= — only the actual service process reads that file. A plain sudo -u openclaw openclaw config validate run by hand won't have the key in its environment even though the
running service does, and will report the provider as unavailable for what looks like a config
reason but is really a missing-credential-in-this-shell reason. To test as the CLI would see it in
production, source the env file and fix HOME explicitly (sudo -E alone doesn't fix HOME when
switching from root):
sudo bash -c 'set -a; source /root/.config/openclaw-*.env; set +a; \
sudo -u openclaw HOME=/home/openclaw openclaw config validate'
Verify a tool actually fired, don't trust "service restarted clean." openclaw agent --agent <id> -m "<message that forces the tool>" --json runs a real turn through the live gateway
(omit --local to use the actual running service, not an embedded one-off) and returns a
toolSummary: { calls, tools: [...], failures } field — concrete proof a specific tool executed,
plus the model's own reply usually names the provider it used. This is the same "verify the
actual feature, not just connectivity" lesson as the loopback-firewall incident below, applied to
a new feature instead of a new firewall.
Exec sandboxing interacts badly with a host-level uid firewall
agents.defaults.sandbox (Docker/Podman-based exec isolation) puts sandboxed commands in their
own network namespace. If you've also built a host-level outbound firewall scoped by uid (e.g.
nftables meta skuid <id> rules) to restrict what the service account can reach, that firewall
will not apply to sandboxed exec calls — Docker's networking bypasses it entirely unless you
build a second, separate firewall for the Docker network (e.g. DOCKER-USER iptables chain
rules). This is a general Docker-vs-ufw/host-firewall gotcha, not OpenClaw-specific, but it's
easy to only discover once you've built both pieces separately and assumed they compose.
Given security: "full" + ask: "always" already means nothing executes without a live human
decision, treat Docker sandboxing as an additional defense-in-depth layer to add deliberately
later (with its own matching network firewall), not something to enable casually alongside a
host-level firewall and assume both apply.
Uid-scoped outbound firewall (composes correctly with ufw)
To restrict what the service account can reach outbound (e.g. only the LLM/messaging/deploy
APIs it actually needs, plus one explicit SSH carve-out to a specific internal host) without
touching ufw's own tables: add a separate nftables table at a later priority than
ufw's default (filter + 10 composes correctly after ufw's priority 0 — nftables
processes all base chains at a given hook across all tables, in priority order; ufw's ACCEPT
at priority 0 doesn't terminate the hook, so your later-priority DROP still gets evaluated):
table inet openclaw_egress {
chain output {
type filter hook output priority 10; policy accept;
oifname "lo" accept
meta skuid 1002 tcp dport 443 accept
meta skuid 1002 udp dport 53 accept
meta skuid 1002 tcp dport 53 accept
meta skuid 1002 tcp dport 22 ip daddr <trusted-host-ip> accept
meta skuid 1002 counter drop
}
}
The oifname "lo" accept line is not optional. OpenClaw's own gateway process connects to
itself over loopback (127.0.0.1:<gateway-port>) for its internal exec-approval handler — this
is not exposed traffic in any meaningful sense (it never leaves the host), but a uid-scoped
egress rule with no loopback exemption drops it exactly like any other unlisted destination.
Symptom: exec calls silently hang forever with no approval prompt ever reaching Telegram, no
error surfaced to the user, and only an occasional [telegram] failed to start native approval handler: Error: connect ETIMEDOUT 127.0.0.1:<port> in the gateway's own logs as a clue — easy to
miss since the gateway otherwise looks completely healthy (systemctl status active, Telegram
polling running, model API calls succeeding). Confirm via nft list table inet openclaw_egress
— a nonzero packet counter on the final drop rule while loopback connections are failing is
the tell. After building this firewall, always re-run a live end-to-end exec-approval test
(not just the external allow/deny checks) — the loopback break only shows up when something
actually tries to invoke exec, not from checking connectivity to external hosts.
Never nft flush ruleset to apply this (wipes ufw's and tailscaled's own tables too — see
the homelab's general Tailscale/nftables gotcha). Use nft delete table inet openclaw_egress 2>/dev/null; nft -f <file> — scoped to just this one table, safe to re-run. Persist via a
systemd oneshot unit (ExecStart= that same delete-then-load line, RemainAfterExit=yes), not
nftables.service//etc/nftables.conf if ufw isn't already using that mechanism on the host
(check systemctl is-active nftables first — if ufw is managing its own tables independently,
adding your rules into the same lifecycle is unnecessary and can cause confusion about which
mechanism owns what).
Verify both the allow and deny cases explicitly, as the target uid — a passing curl to an allowed host doesn't prove the deny side works:
sudo -u openclaw curl -s -m 5 -o /dev/null -w '%{http_code}\n' https://api.anthropic.com/
sudo -u openclaw timeout 5 bash -c 'echo > /dev/tcp/<some-other-internal-host>/22' \
2>&1 && echo "BAD: connected" || echo "good: refused"
Telegram channel lockdown
Numeric user ID only — Telegram usernames/phone numbers are not accepted in allowFrom. Get
your own ID via /whoami@<bot_username> in a DM, or from the pairing request record.
Initial setup uses dmPolicy: "pairing" (generates a one-time code on first DM), approved via:
openclaw pairing list telegram
openclaw pairing approve telegram <CODE>
The first approval auto-bootstraps commands.ownerAllowFrom if empty. This does not fully
lock the channel down by itself — anyone can still trigger a new pairing request (even if
they can't get approved) while dmPolicy stays "pairing". For a single-owner personal
instance, switch to a hard allowlist afterward so no one else can even initiate pairing:
{
channels: { telegram: { dmPolicy: "allowlist", allowFrom: ["<your-numeric-id>"] } },
commands: { ownerAllowFrom: ["telegram:<your-numeric-id>"] }
}
Skills: prefer first-party bundled or local-authored over ClawHub
openclaw skills search <name> queries ClawHub, an open, unvetted community marketplace —
results routinely include several near-duplicate submissions from different authors under the
same apparent name, and outright spam/SEO-farming content mixed in. Before installing anything
from there, check whether a first-party bundled skill already covers it — these ship with
OpenClaw itself (Source: openclaw-bundled in openclaw skills info <name>) and just need a
binary requirement satisfied to go from △ needs setup to ✓ ready:
openclaw skills list # shows bundled skills gated on missing binaries
openclaw skills info <name> # confirms Source: openclaw-bundled vs. clawhub/other
Example: the bundled github skill needs only the official gh CLI installed — no ClawHub
install required at all, and it's first-party-trusted rather than arbitrary community code.
For anything with no bundled equivalent (e.g. Cloudflare, as of this writing), author a small
local SKILL.md yourself instead of pulling an unreviewed ClawHub entry — same frontmatter
shape as bundled skills, installed via:
openclaw skills install <local-dir> --as <slug>
This keeps every piece of skill code either first-party or self-authored, with only credentials (scoped API tokens) as the actual trust boundary being extended to third parties.
Dashboard / Control UI
openclaw dashboard opens a web Control UI — but if the gateway is bind: "loopback" (the
right choice per OpenClaw's own security docs for anything with real infra access), it's only
reachable from the box itself at http://127.0.0.1:<port>/. From a workstation, either SSH
port-forward (ssh -L <port>:127.0.0.1:<port> user@host, then open the URL locally) or expose it
deliberately via the same reverse-proxy pattern used for everything else (Tailscale Serve on its
own port, not the loopback bind). Get the real bound port and dashboard URL from:
openclaw gateway status --deep
This same command is the right first diagnostic for "the CLI can't connect to my running
gateway" — it separates "gateway process isn't running" from "gateway is running but the CLI's
own auth/probe handshake is failing" (the latter showed up as a Connectivity probe: failed / timeout even with the gateway confirmed running and correctly bound — a CLI-context auth
issue, not a service-health issue; don't treat it as proof the gateway itself is unhealthy).
Logging into the dashboard from a browser takes two more steps beyond just the URL:
openclaw config get gateway.auth.tokenprints__OPENCLAW_REDACTED__, not the real value — the CLI deliberately masks secrets in its own terminal output. Read the raw config file instead:python3 -c "import json; print(json.load(open('~/.openclaw/openclaw.json')) ['gateway']['auth']['token'])"(expand~or use the real path).- The Control UI needs its own device-pairing approval, separate from any channel (Telegram
etc.) pairing already done — first connection attempt shows "Device pairing required" with a
request ID. Approve it the same way as a channel pairing, just a different command:
openclaw devices list/openclaw devices approve <request-id>.
Secrets: two valid patterns, pick per-field based on what's documented
SecretRefsourced from env (for values wired through the systemd unit'sEnvironmentFile=):{ "source": "env", "provider": "default", "id": "ANTHROPIC_API_KEY" }- Plaintext directly in
openclaw.json(for values with no natural env-var home, like the gateway's own auth token) — OpenClaw's own docs explicitly say to protect this via file permissions (chmod 600 ~/.openclaw/openclaw.json,chmod 700 ~/.openclaw/), confirming this is an intended, not accidental, pattern.
Generate any value that needs to land in either place server-side, in one script, so it never transits through your own shell history or an assistant's chat transcript.
Evaluating ClawHub skills/plugins by real usage, not vibes
openclaw skills search <query> --json and openclaw plugins search <query> --json return real
metrics per result: stats.{installs,downloads,stars}, isSuspicious, and for plugins
channel (official/community) + isOfficial. Use these to rank, don't guess from
descriptions. Two real findings from a 2026-08-16 survey across ~15 categories on dfw:
- The "plugins" (code-level) family is a much thinner ecosystem than "skills" (CLI-wrapper
style). Outside
@openclaw/-official entries (Brave, LanceDB memory), almost nothing broke 35 installs across a dozen categories checked. Real adoption numbers (500-1500+ installs, dozens of stars) live in the skills family — Slack, PDF, Calendar, Linear, etc. Don't recommend a "plugin" for a capability that has a well-adopted "skill" equivalent instead. - High install/star counts and
isSuspicious: falsedon't mean the skill is well-built or even a clean fit — always read the actualSKILL.md/scripts, not just the search-result summary, before installing something security-relevant or expecting a specific behavior:ivangdavila/translate(381 installs, 10★): not a standalone translator — it's a thin skin over a separate third-party product ("Clawic", clawic.com) that expects a whole personal-data directory tree (~/Clawic/data/contacts/,~/Clawic/data/projects/,~/Clawic/data/finances/,~/Clawic/profile.yaml) before it'll even activate (Visible to model: nountil then). Adopting it means adopting a second personal-assistant memory system, not just getting translation. Skip unless that tradeoff is deliberately wanted.cclank/news-aggregator-skill(349 installs, 23★): summary says "8 major sources" but the actual list (readscripts/fetch_news.py) is Hacker News, GitHub Trending, Product Hunt, plus 36Kr, Tencent News, WallStreetCN, V2EX, and Weibo — mostly Chinese tech/finance/social sources, not general news. If that's not wanted, patch the script's--source alldefault (asources_mapdict keyed by source name) rather than relying on the agent remembering to pass--source hackernews,github,producthuntevery time.adboio/agentmail(1138 installs, 66★ — by far the most-used of 5 competing ClawHub wrappers for the same underlying AgentMail service): its bundled example code was written against an olderagentmailPyPI SDK version. Against the real installed SDK (0.5.9),client.inboxes.create(username=..., client_id=...)throwsTypeError: unexpected keyword argument 'username'— the current SDK wantsclient.inboxes.create(request= CreateInboxRequest(username=..., client_id=...)).send()andlist()still match the documented flat-kwarg style; onlycreate()drifted. Checkinspect.signature(...)against the actually-installed package before trusting a skill's example code verbatim, especially for the first call in a quick-start.- Same skill's "Security: Webhook Allowlist (CRITICAL)" section — the actual defense against
prompt-injection-via-arbitrary-inbound-email — was written for the pre-rename Clawdbot
product and references
~/.clawdbot/hooks/,~/.clawdbot/clawdbot.json, and aclawdbot gateway restartcommand. None of those paths/commands exist on a real OpenClaw install (~/.openclaw/openclaw.json,openclaw gateway restart). Following it as written produces something that looks configured but provides zero actual protection. Grep any security-critical section of a ClawHub skill for stale product-name paths before trusting it — general lesson: a skill's popularity is a signal about the capability being useful, not a guarantee the documentation was ever updated across a product rename or an SDK major version bump.
Voice-note transcription (tools.media.audio)
Inbound voice notes (Telegram etc.) are transcribed by OpenClaw's own tools.media.audio
pipeline, not by installing a "skill" — this is a core feature, currently unconfigured by
default. Auto-detect order when no provider is pinned: Groq → OpenAI → xAI → Deepgram → Google
→ SenseAudio → ElevenLabs → Mistral, then local CLI fallback (whisper-cli/whisper) if no
provider auth resolves. The legacy audio.transcription.command config key still parses but is
retired — use tools.media.audio.models (openclaw doctor --fix migrates old {input}
placeholders, not the key itself).
On a small VPS (1 vCPU/4GB), prefer a cloud provider over local Whisper — local transcription adds real CPU load per voice note plus first-run model-download latency. Groq is a good default: first in auto-detect priority, cheap/fast, generous free tier. Setup:
openclaw plugins install @openclaw/groq-provider # official external package, not bundled
openclaw config set tools.media.audio.models '[{"provider":"groq"}]'
# bare GROQ_API_KEY in the EnvironmentFile is NOT enough for the audio path -- see below
openclaw config set models.providers.groq.apiKey '{"source":"env","provider":"default","id":"GROQ_API_KEY"}'
# add "groq" to plugins.allow if you've locked it down (see the plugins.allow section above)
sudo systemctl restart openclaw.service
The docs say a bare GROQ_API_KEY env var is sufficient ("Auth env var: GROQ_API_KEY"), and
two independent checks appeared to confirm that — both were false positives for the audio path
specifically. curl directly against api.groq.com/.../audio/transcriptions with the key
returned a real 200 + transcript, and openclaw models list --provider groq showed every model
with Auth: yes. Both genuinely passed. A real inbound Telegram voice note still failed with
[media-understanding] audio: failed (0/1) reason=ProviderAuthError, even though GROQ_API_KEY
was confirmed present in the actual running gateway process's environment
(sudo cat /proc/$(systemctl show openclaw.service -p MainPID --value)/environ | tr '\0' '\n'
— use the real systemd MainPID, not pgrep -f 'openclaw gateway', which can match a stray
interactive CLI invocation and silently check the wrong process's environment entirely). The
models list check only exercises chat-model auth resolution, not the separate audio
media-understanding auth path — a passing chat-model check doesn't prove the audio path will
resolve auth the same way. Fix: add an explicit models.providers.groq.apiKey SecretRef (same
pattern already used for anthropic in a working config) rather than relying on bare env-var
auto-pickup for the audio path. Confirmed fixed via a real voice memo after adding it.
There's no CLI flag to attach a media file to a test turn (openclaw agent --help has no
audio/attachment option), so the curl+models list checks above are the best CLI-only
verification available — but given the false-positive above, treat them as necessary, not
sufficient. A real voice memo through the actual channel is the only thing that fully exercises
OpenClaw's own attachment-download-and-inject glue plus the audio-specific auth path.
Memory indexing (openclaw memory index) needs its own embedding provider auth
Same theme as the audio path above: memorySearch.provider defaults to "openai", entirely
independent of whatever chat-model provider is configured (Anthropic, in a typical build here).
Configuring ANTHROPIC_API_KEY for chat does nothing for memory embeddings — openclaw memory index --force --agent <id> fails with No API key found for provider "openai" until an OpenAI
(or another explicit memorySearch.provider) credential exists. This is the third distinct
instance of "OpenClaw subsystems each need their own separate provider auth" (see model-selection
and Groq-audio notes above) — always check openclaw memory status --index --agent <id> (Auth store: line names the exact sqlite file, Provider:/Embeddings: lines show real state) rather
than assuming a working chat model implies memory search works too.
Fix: get an OPENAI_API_KEY, scoped narrowly (OpenAI dashboard → Restricted → expand Model
capabilities → set only Embeddings (/v1/embeddings) to Request, everything else None —
this key never needs chat/completions/files/etc access), add it to the same EnvironmentFile as
the other provider keys, restart openclaw.service, then reindex:
sudo bash -c "echo 'OPENAI_API_KEY=sk-...' >> /root/.config/openclaw-anthropic.env"
sudo systemctl restart openclaw.service
sudo bash -c 'set -a; source /root/.config/openclaw-anthropic.env; set +a; \
sudo -u openclaw --preserve-env=OPENAI_API_KEY,ANTHROPIC_API_KEY HOME=/home/openclaw \
openclaw memory index --force --agent main'
Non-Ollama/non-local alternatives (Bedrock, DeepInfra, Gemini, Mistral, Voyage,
openai-compatible) and the fully-local provider: "local" (GGUF via llama.cpp, no API key,
~0.6GB auto-downloaded default model) are documented in the package's own
/usr/lib/node_modules/openclaw/docs/reference/memory-config.md — check the bundled copy, it's
far more complete than the public docs site (same lesson as elsewhere in this file). ollama
appearing in plugins.allow does not mean an Ollama daemon is actually installed/running —
that's just the bundled provider-adapter code; verify with which ollama /
systemctl is-active ollama before assuming the local-embeddings path is available on a given
box, especially a small VPS where standing up a whole second model-serving daemon has a real
resource cost worth weighing against a fractions-of-a-cent-per-month OpenAI embeddings bill.
memorySearch.extraPaths indexes an arbitrary external directory of markdown files — separate
from, and easy to conflate with, conversation/session indexing. sources: ["memory","sessions"]
(the more commonly documented option) indexes OpenClaw's own conversation history. extraPaths
(array of absolute or workspace-relative paths, same bundled memory-config.md reference) is a
different feature entirely: point it at a directory and it recursively indexes every .md file
found there as external knowledge — e.g. a synced Obsidian vault, a docs folder, notes a human
maintains by hand. Functionally the OpenClaw analog of Claude Code's autoMemoryDirectory, but
the consumption model differs even when both point at the same files: this is a chunked,
embedded, top-K-retrieved-per-query index (hybrid BM25+vector), not a curated set of files read
wholesale into context every turn. Same source notes, not guaranteed same "knowledge" from a
given note. Confirmed real and current 2026-08-18 via the bundled doc — not yet built/exercised
on any live instance, so treat the mechanics above as verified-on-paper, not verified-in-use.
Redaction gotcha hit while debugging a bad key-append attempt: when eyeballing an env file's
structure without printing secret values, a naive grep -oE '^[A-Za-z_]+='-style check only
shows lines that match the expected NAME=value shape — a malformed line (e.g. someone pasted
just the raw secret with no NAME= prefix, dropped when substituting into a copy-paste command
template) silently doesn't match and won't show up as an anomaly, but also won't get redacted if
you then try to eyeball "what's really in this file" with a sed/awk substitution that only
transforms matched lines — every unmatched line prints verbatim, secret value and all. Confirmed
the hard way: a bare unprefixed OpenAI key leaked into tool output this way and had to be
revoked/rotated. Safe pattern is to make truncation unconditional regardless of match, e.g.:
sudo awk '{ if (match($0, /^[A-Za-z_]+=/)) print substr($0,RSTART,RLENGTH) "[...]"; \
else print "<unnamed line, " length($0) " chars>" }' /root/.config/openclaw-anthropic.env
so a malformed line still reveals that it's wrong (and its length) without ever printing its
content. See the global CLAUDE.md's "never cat-then-redact" gotcha — this is the same failure
mode, just via a pattern-matched redaction script instead of a raw cat.
Non-chat provider use is invisible to a model-fetch journal grep (2026-09-12)
The two subsystems above (Groq audio transcription and OpenAI memory_search embeddings) don't
emit model-fetch journal lines. An audit that counted provider=openai/provider=groq model-fetch
lines over 7 days found zero and nearly retired both keys. The agent then checked its live config:
embeddings ran on every memory recall, and voice notes were being transcribed through Groq days earlier.
Before calling any provider key unused, check every capability it can back:
models.providers, the model and fallback chains, and per-cron overridestools.media.models(audio, image)memorySearch.provider, and whether its default appliesplugins.allow/ enabled provider plugins
The service account's config is usually off-limits to the operator, so ask the agent to check its own live config. Removing a key also needs its capability re-pointed first. An unconfigured capability in the media fallback chain can silently bill a more expensive provider.
PEP 668 (EXTERNALLY-MANAGED) blocks pip for Python-based skills too
Some ClawHub skills (AgentMail, others with Python SDKs) need pip install <package> to actually
work. Debian 13's system Python refuses both plain pip install and pip install --user under
PEP 668 (error: externally-managed-environment) — same root cause as the Mac skill-creator
gotcha in the global CLAUDE.md, different fix since this is a dedicated single-purpose host
rather than a shared dev machine. Pattern that keeps plain python3 calls from the agent's exec
tool working transparently (no venv-activation ceremony needed in every script):
sudo apt-get install -y python3.13-venv # not installed by default; ensurepip fails without it
sudo -u openclaw python3 -m venv /home/openclaw/.venvs/base
sudo -u openclaw /home/openclaw/.venvs/base/bin/pip install agentmail python-dotenv
Then prepend the venv's bin/ to the service's PATH (same mechanism already used for
Homebrew — see the Homebrew note above) so python3 resolves to the venv, not system Python, for
every exec-spawned child of the gateway process:
Environment=PATH=/home/openclaw/.venvs/base/bin:/home/linuxbrew/.linuxbrew/bin:...(existing PATH)
sudo systemctl daemon-reload && sudo systemctl restart openclaw.service to apply. Verify with
the exact PATH the service will see (not just an interactive shell, which may differ):
sudo -u openclaw env -i HOME=/home/openclaw PATH='<paste the exact Environment=PATH value>' \
bash -c 'which python3; python3 -c "import agentmail; print(1)"'
Inbound webhooks (hooks.mappings) — the real delivery mechanism, and a recurring corruption bug
Built 2026-08-16: Cloudflare Email Routing → a Worker (email() handler, envelope
message.from — not spoofable, unlike a header From:) → POST to OpenClaw's hooks endpoint
→ hooks.mappings → agent turn → Telegram. Config:
{
hooks: {
enabled: true,
token: "<dedicated random token, NOT the gateway auth token>",
path: "/hooks-<random-suffix>", // non-guessable path + token together
allowedAgentIds: ["main"],
mappings: [{
id: "mail-inbound",
match: { path: "mail" }, // RELATIVE to hooks.path -- NOT the full path.
// match.path: "/hooks-xxx/mail" 404s; "mail" works.
action: "agent",
sessionKey: "telegram:direct:<chat-id>", // see below -- do NOT use a separate hook session
messageTemplate: "...{{messages[0].from}}...{{messages[0].subject}}...{{messages[0].snippet}}",
deliver: true,
channel: "telegram",
to: "<chat-id>",
allowUnsafeExternalContent: false, // keep false even for allowlisted senders --
// body content is untrusted *information*, not
// trusted *instructions*
}],
},
}
Sender trust boundary belongs at the Worker, not the transform. The Worker checks the SMTP
envelope message.from against an explicit allowlist and silently returns (no setReject(),
so probing senders can't confirm the address exists) before ever forwarding. hooks.mappings
templates (messageTemplate/textTemplate, {{messages[0].field}} syntax) are sufficient for
shaping the payload — there is no worked example of the transform.module/export JS-function
contract anywhere in the bundled docs, so don't guess at it for something handling untrusted
input; use templates against a payload shape you control instead (have the sender-side Worker
emit {"messages":[{"id","from","subject","snippet"}]} directly).
deliver/channel/to did not work as documented — the real fix was session targeting
Setting deliver: true, channel: "telegram", to: "<chat-id>" had zero effect on where the
reply actually went, across many permutations (to as bare chat ID, "direct:<id>",
channel: "telegram" vs the Gmail example's channel: "last"). In every case the agent's own
turn used a sessions_send tool call to relay into a session literally named main
(agent:main:main) — visible in the Control UI as "Forwarded from main" — regardless of the
mapping's delivery config. This looks like agent-driven behavior (the model choosing to relay
via sessions_send to what it treats as the canonical session), not something the mapping's
channel/to fields control for action: "agent".
The fix: point sessionKey directly at the real interactive channel session, not a separate
hook-scoped key. Find the real session key from openclaw sessions --json --active <n> (look
for kind: "direct" bound to the real channel, e.g. agent:main:telegram:direct:<chat-id> —
the part after agent:<agentId>: is what hooks.mappings[].sessionKey takes). With
sessionKey: "telegram:direct:<chat-id>", the hook-triggered turn runs inside the same
session as the user's normal conversation — no separate hook session, no cross-session relay,
no dependency on whatever main happens to be. Matching hooks.allowedSessionKeyPrefixes and
hooks.defaultSessionKey need the same real prefix (e.g. ["telegram:"]), not a made-up
"hook:" prefix.
Do not trust openclaw config validate for this class of bug. It only does static schema
checks — hooks.allowedSessionKeyPrefixes must include 'hook:' when hooks.defaultSessionKey is unset is a startup-time cross-field check that validate does not catch, and a bad value
here crash-loops the entire gateway (see below), not just the hooks feature.
A cron job's sessionTarget pointed at the live conversation session can queue behind it and time out — caveat to the fix above
The section above recommends pointing a hook's sessionKey directly at the real interactive
channel session to avoid a separate hook-scoped session. The same pattern applied to a cron
job's sessionTarget (e.g. "session:agent:main:telegram:direct:<chat-id>") has a real
downside: that session is a single lane, and a cron-triggered turn competing with an active
turn in the same lane (a live conversation in progress) queues behind it rather than running
concurrently.
Confirmed 2026-08-30/31 (LunaRoute pilot, weekly-share-more-nudge cron job): every forced test
run failed with TimeoutError: cron: isolated agent setup timed out before runner start (a fixed
~60s wait) while debugging was actively happening in that same session. journalctl -u openclaw.service showed the real cause directly: lane wait exceeded ... activeAhead=1 right
before the timeout — the cron turn was queued behind the live conversation's own in-flight turn,
not failing for any provider/config reason. This looked at first like a provider-specific bug
(the pilot was mid-switch to a new model provider at the time) and cost real debugging time before
the lane-contention signature was recognized in the logs.
Fix: for a cron job whose turns don't need to share context/history with a live conversation
(the common case — most cron jobs are self-contained, not conversational continuations), set
sessionTarget: "isolated" instead of pointing it at the interactive session. This is also
already the pattern other cron jobs in this environment use. A job that does need to run inside
the same session as live chat (relaying into an ongoing conversation, similar to the hooks case
above) will still hit this queuing behavior any time it fires mid-conversation — that's an
inherent tradeoff of sharing the lane, not a bug to fix, so don't pick shared-session targeting
for a cron job without weighing this.
A specific session key can get "stuck" independent of the transcript file
agent:main:main returned FailoverError: Unknown model: anthropic/claude-sonnet-5 on every
turn, for many minutes, while every other session (fresh hook sessions, the real Telegram
session, plain CLI tests) used the identical model string successfully. Diagnosis path that
worked: openclaw sessions --json --all-agents --limit all, find the session's status field
("failed" — separate from whether its .jsonl transcript file was actually written to;
stat-checking the transcript file's mtime showed it predated the failures entirely, proving
the "failed" status lives in separate session-store metadata, not the transcript). Retrying the
exact same session key even with a fresh sessionId (confirmed via the session list) failed
identically — ruling out simple stale per-session state and pointing at something specific to
that session key or its bootstrap path (this session had a much larger skillsSnapshot than
hook sessions, i.e. a full "main" agent bootstrap vs. a lighter hook-triggered one).
The actual fix was a full systemctl restart openclaw.service — not a config change, not a
targeted per-session repair (no CLI command exists for the latter: openclaw sessions only has
cleanup/compact/export-trajectory/tail, nothing that resets a stuck status). This
strongly suggests the bug is in-memory/live-process state, not anything persisted to disk.
This corruption recurred multiple times over one gateway process's lifetime, and it spread
— it started isolated to agent:main:main, then later the same error hit a previously-healthy
hook session (agent:main:hook:mail:inbox) and lane=cron-nested in the same burst. The
trigger appears to be repeated openclaw config set hot-reloads within one process run —
each one logs [reload] config change detected... hot reload applied, and this session made
dozens of them in under an hour while iterating on the hooks config. Lesson: past a handful of
hot-reload cycles in one debugging session, if a different error shape appears (e.g. a
"channel routing" bug suddenly presents as "Unknown model"), treat that shape change as the
signal to restart immediately rather than continuing to iterate on config — a changed error
class is a stronger "try the cheap reset" signal than a repeated one. Prefer batching remaining
config edits into one --batch-file apply + single restart over many incremental config set
calls once a session has already needed one corruption-clearing restart.
Recurred again, 2026-08-23, different session — same signature, same fix, worth trusting the
pattern-match over re-investigating from scratch. A one-shot cron job (lane=cron-nested,
sessionKey=agent:main:cron:<id>:run:<id>) failed twice in a row with the identical
FailoverError: Unknown model: anthropic/claude-sonnet-5, in a gateway process that had by
then absorbed a full session's worth of plugin installs, config edits, and multiple prior
restarts (building a cross-agent filesystem channel, installing/testing a before_agent_finalize
hook plugin, several config set-driven hot reloads along the way). Recognized immediately from
the error string alone, confirmed via journal ([reload] config change detected... hot reload applied entries earlier in the same window), fixed the same way — plain systemctl restart openclaw.service, gateway came back with the same plugin list intact, no further recurrence
in the following minutes. Didn't re-diagnose from zero; matching this exact error string against
this section is faster and just as reliable as re-deriving it.
Tailscale Funnel to a same-node Serve chain: use single-hop --set-path, not two hops
tailscale funnel --bg 8445 (targeting an existing tailscale serve --https=8445 ... mount on
the same node) produced a 502 from the public URL while the same request against the
tailnet-scoped :8445 URL directly returned 200 — the two-hop chain (Funnel→Serve→gateway) had
a working Serve leg but a broken Funnel leg. Root cause not fully diagnosed (likely a TLS/backend
protocol mismatch specific to chaining Funnel into another local HTTPS-terminating Serve port),
but the fix sidesteps it entirely: funnel supports --set-path directly, same as serve.
Collapse to one hop instead of chaining:
tailscale serve --https=8445 off # tear down the two-hop chain if already built
tailscale funnel --bg --set-path=/hooks-xxx http://127.0.0.1:18789/hooks-xxx
This also fixes a secondary problem: an earlier tailscale funnel --bg 8445 (no --set-path)
put a wildcard / mount on port 443, publicly exposing the entire gateway (dashboard
included) rather than just the intended path — caught by testing the bare root URL from an
off-tailnet client (curl -o /dev/null -w '%{http_code}') and seeing something other than a 404.
Always verify the bare root 404s (nothing else mounted) in addition to testing the actual
endpoint, for any Funnel exposing only one path of a multi-purpose gateway.
The CLI syntax changed from older docs/muscle memory: tailscale funnel --bg <port> on now
errors (the CLI for serve and funnel has changed) — current form drops the trailing on
(tailscale funnel --bg <port>), and tailscale funnel status/tailscale serve status (not
--json-only) are the fastest way to see the live real mount table.
Verifying against a real inbound event needs to account for mail delivery latency
A curl straight to the gateway's hooks endpoint proves the OpenClaw-side pipeline; it does not
prove the Cloudflare→Worker leg. For that, either watch wrangler tail --format pretty live
(each real email shows as Email from:X to:Y size:N @ <time> - Ok plus the Worker's own
console.log) or check openclaw journalctl for real activity. First-time delivery to a
brand-new MX/routing destination is commonly delayed by several minutes (greylisting-like
behavior on the sending side is a plausible explanation, not confirmed) — a wrangler tail
session that expires (60-180s) or a narrow journalctl --since window checked too early will
both show nothing even though the email is genuinely in transit and arrives moments later.
Don't read "nothing captured in my tail window" as "the pipeline is broken" without first ruling
out a timing mismatch between when you're watching and when delivery actually completes —
re-arm a fresh watch and wait longer before concluding infrastructure is at fault, especially
once the known bugs above are already fixed.
A CLI health-check probe can land directly in the user's real, visible conversation
openclaw sessions --json lists session keys like agent:main:main alongside ones you created
for testing — nothing distinguishes "this is a throwaway diagnostic target" from "this is the
user's actual default cross-device session." agent:main:main specifically is the generic
default session used by bare CLI calls and by the web Control UI when accessed directly (not
through a specific channel like Telegram) — labeled "Main Session" in the UI. Running
openclaw agent --session-key agent:main:main -m "health check..." repeatedly to probe whether
the model registry had recovered put that literal text into the user's real conversation
history, visible on every device logged into the Control UI (confirmed via a phone screenshot).
Before using any specific session key as a health-check target, confirm via the Control UI or
by asking the user whether that key is something they actually view — don't assume a
generic-sounding key name is safe to write test content into. Prefer verification paths that
don't inject a message into any conversation at all: openclaw doctor, config validate, or a
models-catalog list.
OpenClaw's own session files are local context only — they don't reach already-sent Telegram messages
Once a hook-triggered (or any) turn's reply has actually been delivered to Telegram, editing or
deleting the underlying OpenClaw session .jsonl file does nothing to what's sitting in the
user's real Telegram chat history — that's Telegram's own server-side record, entirely separate
from OpenClaw's local transcript. To actually remove an already-delivered message, use the
Telegram Bot API directly with the bot's own token:
curl "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/deleteMessage" \
-d chat_id=<chat-id> -d message_id=<id>
Get the real message_id values from the gateway's own logs ([telegram] outbound send ok accountId=default chatId=<id> messageId=<N> ... in journalctl -u openclaw.service) — there's
no way to browse/preview a bot's own sent messages via the API first, so cross-reference by
timestamp against when you know test content was sent, and get the user's explicit go-ahead
before deleting anything from their real chat history (one wrong guess deletes a message you
can't identify to restore). Separately, editing a local session's .jsonl (e.g. to strip test
content while preserving the session's header/system lines) is safe and effective for cleaning
up the web Control UI's own view of a session — back the file up first, remove only the
specific message-type JSON lines by id, and leave header (session/model_change/etc.) lines
intact so the session itself keeps working.
Granting an agent real infra privilege: scoped dispatch scripts, not broad sudo
Built and verified 2026-08-17 when Will asked for OpenClaw (running as "Olu") to be able to do read-only + non-destructive homelab administration — the same category of task a Claude Code session does interactively, but reachable by the agent itself via its own exec tool.
The dispatch script is the security boundary, not the sudoers grant. Don't write a sudoers
file with Cmnd_Alias wildcards (ansible all -m shell -a * is unrestricted root in disguise).
Instead grant NOPASSWD sudo to exactly one root-owned script path per capability tier, and put
all the real scoping logic — host/unit/playbook allowlists, regex-validated arguments, no shell
interpolation — inside that script:
# /etc/sudoers.d/openclaw-remote
openclaw-remote ALL=(root) NOPASSWD: /usr/local/bin/openclaw-fleet-admin-readonly.sh
openclaw-remote ALL=(root) NOPASSWD: /usr/local/bin/openclaw-fleet-admin-changes.sh
Split into two scripts — readonly and changes — as separate files/paths, not one script with
internal risk-level branching. OpenClaw's own exec-approvals.json allowlist matches on
resolved binary path, not arguments, so a single script mixing safe reads and real mutations
can't be partially allowlisted — it's all-or-nothing. Two paths let the path itself carry the
trust decision: add only the *-readonly.sh path to the allowlist (skips Will's Telegram
approval prompt), leave *-changes.sh off it entirely (every invocation still prompts). This
keeps "capability" and "auto-approval" as genuinely separate axes — granting the ability to
restart a service doesn't mean it happens without Will seeing it.
Verify the boundary holds, not just that the happy path works: attempt a raw sudo <arbitrary command> as the granted user (should fail — not in sudoers), an out-of-scope subcommand (should
be rejected by the script's own case/die logic), and a path-traversal argument like
playbook-run ../../etc/passwd (should be rejected by regex validation before it ever reaches
ansible-playbook). All three are meaningful negative tests, not paranoia.
sudoers NOPASSWD matches the literal granted command, not anything that merely invokes the
same script. Confirmed 2026-08-18 auditing Olu's exec failures: sudo bash -x /usr/local/bin/openclaw-fleet-admin-changes.sh ... (debugging with bash -x) and sudo cat /usr/local/bin/openclaw-fleet-admin-readonly.sh (reading the source) both fell through to an
interactive password prompt and failed headless — sudoers saw bash and cat as the command,
not the script path it actually granted. The agent's own retries kept failing the identical way
because the error ("a terminal is required to read the password") looks like a broken sudo
grant, not a wrong invocation shape. Fix is on the calling side, not the sudoers config: the
granted script must be run bare (sudo /usr/local/bin/openclaw-fleet-admin-changes.sh <subcommand> <args>) — never wrapped in bash -x, piped through cat, or prefixed with
anything else, even for debugging.
This sudo pattern works for openclaw-remote over SSH, and CANNOT work for the openclaw service account
An important scoping limit on everything above. The dispatch-script pattern is granted to
openclaw-remote, a separate account reached over SSH — an ordinary login session with no
systemd confinement. The same pattern is impossible for openclaw itself, the account
openclaw.service runs as, because that unit sets NoNewPrivileges=yes: the kernel refuses the
escalation before sudo ever reads sudoers, so any grant to it is decorative. The error is a bare
sudo: The "no new privileges" flag is set, which prevents sudo from running as root, which
reads like a container quirk rather than "this design can never work."
Confirmed 2026-09-04: an agent-email sender shipped to both dfw and hermes as a root-owned
script plus a path-scoped NOPASSWD grant, and had never once run from an agent. It looked
healthy because the only entries in its audit log were the sends root made while testing —
exactly what a working path produces. Olu found it by trying to use it and reporting the
failure, which is the durable lesson: when a mechanism exists so a less-privileged actor can do
something, the acceptance test has to run as that actor under its real confinement. Testing as
root proves the plumbing exists, never that the intended caller can reach it.
systemctl show openclaw -p NoNewPrivileges before designing anything that needs the confined
agent to escalate. If it's yes, invert the privilege boundary instead of widening it: have
the agent write a job file into a spool directory it can already write, and have a root-owned
systemd .path unit collect and act on it. That keeps the credential entirely out of the agent
process, which is usually what the sudo grant was reaching for anyway. Two companions:
ProtectSystem=strict means the spool needs a ReadWritePaths= drop-in and a restart
(applied at process start), and a sudo -u openclaw test from a root shell does not
reproduce the confinement — verify against the running process:
grep /var/spool/agent-email /proc/$(systemctl show openclaw -p MainPID --value)/mountinfo
# want: ... /var/spool/agent-email/queue rw,nosuid,...
See project_agent_email_identity_path memory for the resulting design, and the global
CLAUDE.md entry for the general form.
"Verified live" needs the never-before-exercised branch, not just the idempotent short-circuit
A dispatch subcommand that both mutates state and short-circuits on "already done" (e.g.
dns-add-pihole: skip everything if the record already exists) can pass every "verified live"
test you throw at it while its actual mutation path has never once executed. Found 2026-08-18:
dns-add-pihole's file-edit logic used a Python regex looking for a literal hosts: line that
does not exist in the target playbook (the real list key is pihole_local_dns_hosts:) — so it
could only ever succeed by hitting the already-present shortcut, and would sys.exit() on any
genuinely new record. This shipped and was called "verified live" because the one live test run
(adding dfw) happened to already be present from an earlier manual edit. Lesson: when a
subcommand has an early-exit branch for "nothing to do," the live verification pass must
exercise the other branch too — pick a real, previously-absent input, not a convenient
already-satisfied one — or the mutation code itself is unverified regardless of how confident
the test output looks.
A dispatch script's own argument-validation logic needs the same negative testing as its
mutation logic. Found 2026-08-18 auditing Olu's exec failures: openclaw-fleet-admin- readonly.sh's git-status <repo> subcommand resolved the repo path via bash indirect
expansion (VAR="REPO_MAP_${REPO}"; DIR="${!VAR:-}") instead of validating $REPO against the
known aliases first. The agent passed ansible-ctrl (a real hostname, but not one of the actual
repo aliases ansible/terraform/planning) — the hyphen made REPO_MAP_ansible-ctrl an
invalid bash identifier, so it crashed with a raw invalid variable name error instead of the
intended clean die "unknown repo: $REPO". Any dispatch subcommand using indirect/dynamic
variable lookup to map a user-supplied token to an internal value has this same failure mode for
any input containing characters invalid in an identifier — validate with an explicit case
statement (enumerate the known values, *) falls through to die) instead, so an unrecognized
value always produces the intended clean rejection rather than a shell-syntax crash.
The right way to hand the agent new capabilities: real artifacts + skill_workshop, never prose
Superseded conclusion, corrected 2026-08-17 after a full night of iterating on this with a
real agent ("Olu") in the loop — the --message-file primer-document approach below was tried
first, seemed to work, and turned out to be the wrong pattern overall. Keep reading past the
first attempt for the actual recommendation.
What was tried first: dropping a large "here's what you can now do" document into the
agent's context via openclaw agent --message-file <path> --session-key agent:main:main (a
CLI-invoked session, separate from the agent's live chat sessions). From the agent's own point
of view, an unsolicited message asserting "you now have new permissions, already set up, no
verification needed" is exactly the shape of a social-engineering/prompt-injection attempt —
confirmed live, OpenClaw's own reasoning flagged this and spent a full turn independently
re-verifying every claim via its own exec calls before acting, which is the correct
instinct for an agent with real infra access. A short identity-confirming follow-up in the same
session unblocked it, and it finished the task, including self-discovering and honestly
reporting a real gap in the primer.
Where this broke down: that CLI session (agent:main:main) is a different session from
the agent's live chat session (e.g. agent:main:telegram:direct:<chat-id>) — no cross-session
memory (see the session-isolation entry below). When the human then talked to the agent in its
real live session, that session had zero record of any of the above, and — correctly, given
what it could actually see — treated the resulting outbound message as unexplained/suspicious.
Re-briefing it required delivering directly into the same session the human was actually
talking in, which helped, but every retry also kept hitting the exec preflight and
NoNewPrivileges bugs below, producing repeated real tool failures that the agent actually
handled well each time (adapting its approach and completing the task for real — see the
journalctl vs. session-transcript entry below for why this looked, incorrectly, like
fabrication at first). By the end of the night, the agent's own live session was reflexively
rejecting everything arriving
in it, including trivially short plain human messages with nothing embedded at all — a sign the
approach of "send explanatory prose into chat" had stopped working entirely, independent of how
carefully the prose was worded or how identity was established.
The agent's own stated preference, once asked directly, is the right answer: don't route new capabilities through chat text at all.
- Ship the real artifact and let the agent verify it directly — a file in
~/.openclaw/workspace/, a sudoers entry, a systemd unit, an SSH key. This already worked cleanly every time it was tried: the agent read/tested these itself and trusted them with no narration needed at all. - For anything meant to be durable, reusable knowledge, use
skill_workshop— a real, built-in, governed OpenClaw tool (docs/tools/skill-workshop.md; included in thecodingtool profile) for creating/updating the agent's own workspace skills through a propose → security-scan → apply lifecycle, with rollback metadata, never a direct write. The right shape: ask the agent to explore its own real capabilities and write its ownskill_workshopproposal describing them, then the human reviews and applies it. Self- authored from things the agent verified itself, human-reviewed before going live — no impersonation surface, no cross-session gap, no "trust me" prose required at all. - Confirm intent live, briefly, in plain language, when needed — the agent itself said a short human sentence in the live chat ("yes, that's really new, go check it out") is sufficient; it does not need or want that wrapped in any kind of structured/JSON-looking context block.
If a verification-heavy first turn is still needed (e.g. the very first time a capability is
introduced before this pattern is fully adopted), budget more time than a normal request —
independently confirming several new capabilities via live SSH/exec calls can exceed a 300s
--timeout even with --json output.
NoNewPrivileges=yes silently kills sudo for the agent, regardless of sudoers
If openclaw.service's unit has NoNewPrivileges=yes (real hardening, common and
recommended), granting the openclaw user sudo access to a script via /etc/sudoers.d/
looks correct (sudo -l -U openclaw shows it) but is dead on arrival — the kernel-level
no_new_privs flag blocks sudo's escalation regardless of what sudoers allows, with the
error surfacing only when actually invoked: sudo: The "no new privileges" flag is set, which prevents sudo from running as root. Testing this via sudo -u openclaw bash -c 'sudo ...' from an already-privileged shell does NOT catch the bug — that spawns a fresh
process outside the real systemd unit's confinement, so it succeeds even though the actual
running service never could. To genuinely reproduce the real execution context:
sudo systemd-run --uid=openclaw --gid=openclaw -p NoNewPrivileges=yes --wait --pipe bash -c 'sudo -n true' — this fails the same way the real service does, catching the bug before
trusting the capability.
Fix without relaxing the service's own hardening: don't flip NoNewPrivileges=no on
openclaw.service (weakens sandboxing for everything, not just the one grant needed). Build
a separate systemd service — its own unit, no NoNewPrivileges restriction since it
doesn't need one — that listens on loopback with bearer-token auth and dispatches to the
already-existing scripts. The confined openclaw process then just does a plain
unprivileged curl with the token (no escalation attempt at all), sidestepping the kernel
restriction entirely rather than fighting it. Give the confined process the token via
openclaw.service's existing EnvironmentFile (inherited by any real child process it
spawns) — verify it's actually present with grep -c '^TOKEN_VAR_NAME=' /proc/$(systemctl show openclaw.service -p MainPID --value)/environ
rather than assuming the EnvironmentFile edit took effect.
Verify the caller, not just the callee, for any new SSH-based capability
Testing a new SSH-reachable capability from the receiving host's own local shell (e.g. `su
- -c '...'
run directly on the target) only proves the target-side config is correct — it exercises none of the actual outbound connection. A first real attempt from the genuine calling side can fail immediately onHost key verification failedif that identity has simply never connected before (no entry in its ownknown_hosts) — an easy, easily-missed gap distinct from any sudoers/capability logic. Fix withssh-keyscan -H >> ~/.ssh/known_hosts` (safe for a first-time connection to an already-trusted internal host — this isn't the same risk category as a changed key on a previously-known host, which does warrant independent verification before trusting).
An outbound message from one session looks like unexplained/injected content in another
OpenClaw sessions are isolated by session key (agent:main:main vs.
agent:main:telegram:direct:<chat-id>, etc.) — each has its own separate context/history, with
no cross-session memory of reasoning, even though a tool call in one session (e.g. sending a
Telegram message) produces a real, visible artifact in a channel another session also has access
to. Confirmed live: briefing Olu via --session-key agent:main:main (a CLI-invoked session) led
that session to verify some new capabilities and push a summary out via its Telegram-send tool.
When the user then talked to Olu in the live Telegram session (a different session key), that
session had zero record of the primer, the verification work, or ever deciding to send that
message — from its own context, it looked exactly like a fabricated/injected reply attributed to
itself. This is not paranoia malfunctioning and not an injection — it's genuine session
isolation working as designed, and the agent's skepticism in that moment is correct: it
shouldn't trust an unexplained message just because a human relays "yes, that's real." The fix
isn't to convince the live session to trust it secondhand — point it at a way to verify
independently in its own context (re-read the source file if one exists, or re-run the same
checks itself) rather than trying to argue away the correct instinct.
Telegram's own reply/quote feature can look exactly like a prompt injection to the agent
Confirmed root cause (the agent verified this itself against real source, not just a working
theory) for a real, escalating incident where an agent rejected every subsequent message in a
Telegram thread as a "fabricated internal-context block," including trivially short plain human
text ("will do") with nothing embedded at all — a pattern that had stopped correlating with
message content entirely: replying to (quoting) a specific prior message in Telegram, rather
than typing fresh, makes OpenClaw surface the quoted context to the model wrapped in a real,
built-in envelope (<<<BEGIN_OPENCLAW_INTERNAL_CONTEXT>>>, carrying identifiers including
OPENCLAW_INTERNAL_CONTEXT and inbound_event_kind). This is genuine, documented OpenClaw
behavior (docs/tools/acp-agents.md; shipped in dist/internal-runtime-context-*.js and
dist/get-reply-*.js) — not anything injected. An agent that's never encountered this specific
envelope before, especially one already primed by a genuinely-suspicious delivery earlier in the
same session, can pattern-match "structured internal metadata I don't recognize" onto it and
reject it as hostile, then keep repeating that same judgment on every later turn without
re-verifying against the real source.
The fix, and the general lesson: when an agent flags something as injection-shaped but the surrounding facts don't add up (e.g. it's happening on content-free trivial messages too), have it check its own actual installed source — bundled docs and shipped code — for the literal envelope/tag names it's seeing, before concluding anything is fabricated. In the real incident this section documents, doing exactly that resolved it in one turn, and the agent explicitly, plainly corrected itself afterward, unprompted — good behavior worth recognizing, not just a bug that happened to get fixed. Practically: avoid Telegram reply-quotes to an agent's own messages until this has been established as expected/trusted, or expect the first one encountered to need this exact self-verification step.
exec preflight: complex interpreter invocation detected — write a script file, don't inline it
OpenClaw's exec tool has a built-in safety check (documented in docs/tools/exec.md) that
inspects commands for "common Python/Node shell-syntax mistakes" and refuses to run anything it
flags as a complex inline interpreter invocation — e.g. python3 -c "..." one-liners, or a
curl call with an embedded JSON body and nested quoting — with the error naming the fix
directly: Use a direct 'python <file>.py' or 'node <file>.js' command. This is easy to trigger
by accident when handing the agent example commands that look exactly like what a human would
paste into their own terminal (multi-line, quoted JSON payloads, inline -c snippets) — every
such example in a real incident failed this preflight check, silently costing the agent's entire
turn before it ever got to the actual verification. Per the docs, preflight only inspects files
inside the effective workdir and is skipped entirely when security=full and ask=off —
neither condition held in the config that hit this. When handing off any command more complex
than a single flat binary invocation, write it to an actual file first and tell the agent to run
it as python <file>.py/node <file>.js, rather than a shaped one-liner.
journalctl -u openclaw.service only reliably shows tool-call failures, not successes — check the session .jsonl for the real record
A real incident, worth remembering precisely because the wrong conclusion was drawn from it
initially: a turn reported (with stopReason: stop, a clean finish) as having sent a Telegram
message and an email was checked against journalctl -u openclaw.service for that exact time
window, which showed several [tools] exec failed lines (real — exec preflight blocked two
inline-rendering attempts) but no corresponding success lines for anything — no
[tools] exec ok, nothing about the email. That absence was read as evidence the completion
report was fabricated, and relayed onward as fact. It was wrong. Checking the session's own
.jsonl transcript directly (~/.openclaw/agents/<agent>/sessions/<id>.jsonl, per the
tool-call parsing pattern below) showed the full real sequence: the agent hit the preflight
block, correctly adapted by writing the render step out to a real script file and running it as
python3 <file>.py exactly as the preflight error instructs, and completed both sends for
real, with genuine API responses ({"ok":true,"messageId":"193"} for Telegram,
{"success":true,"result":{"message_id":"<...>@jackson2w.dev"}} for email) sitting right there
in the transcript the whole time.
The actual lesson: journalctl for this service surfaces failures prominently (they're
one-line, alertable events) but does not reliably surface successful generic exec/tool
results at the same log level — likely deliberate (full command/response bodies would be
verbose and can contain credentials, matching the "raw_params":{"reason":"exec command may contain credentials"} redaction already seen on failure lines too). The session .jsonl
transcript is the only complete, authoritative record of what a turn actually did — journalctl
alone is not sufficient to disprove (or confirm) a completion claim. Before concluding an
agent fabricated a result, check the real transcript first, not just the service log.
--message-file bypasses the sandboxed read tool — and can deliver into the wrong directory
openclaw agent --message-file <path> reads the file locally, outside OpenClaw's own tool
sandbox, and injects its contents directly as the turn's user-message text — this works even if
the path sits outside ~/.openclaw/workspace/, since it's the CLI host process reading the file,
not the agent's own read tool call. This creates an easy-to-miss asymmetry: the first delivery
of a document can succeed via --message-file regardless of where the file lives, but if the
agent later tries to re-read that same file itself (e.g. to re-verify content from a fresh
session, per the isolation gotcha above), its own read tool enforces the workspace-only
restriction and will refuse a path outside ~/.openclaw/workspace/ — a Permission denied-style
refusal that has nothing to do with the content being untrustworthy. Write any document meant to
be handed to the agent (via --message-file or otherwise) into ~/.openclaw/workspace/ (a
handoffs/ subdirectory works well) from the start, owned by the openclaw user, so both the
initial delivery and any later independent re-verification use the same accessible path.
Reading a session's .jsonl transcript for tool-call detail
The per-line type field is "message", not the tool name — filter on d["message"]["role"]
and iterate d["message"]["content"] (a list). Assistant tool invocations show up as
{"type": "toolCall", "toolName": ..., ...} (not tool_use/name, despite that being the
Anthropic API's own field naming) alongside {"type": "thinking"} blocks; results arrive in a
following "role": "toolResult" entry. Large tool-call payloads (schemas, big text blocks) are
sometimes stored content-addressed (a *Hash/*Chars pair) rather than inline — don't assume a
missing/empty-looking field means the call had no real arguments before checking whether the
session format hashed it instead.
A tool call mid-turn plus a literal NO_REPLY follow-up silently discards a real, already-generated reply
Real incident, 2026-08-23 on dfw ("Olu"): two inbound Telegram voice notes in a DM got no
reply at all — journalctl -u openclaw.service showed two Anthropic API calls per turn
(vs. one for a plain text turn in the same window), ending in
[turn/kernel] visible channel turn dispatched with no queued reply payloads. Initial
theories (a context-free isolated per-message session; the audio-transcript
[Audio transcript (machine-generated, untrusted)]: wrapper — see the voice-transcription
section above — inducing reflexive silence) were both wrong, and only ruled out by reading the
session's own raw .jsonl transcript directly (see the parsing section just above) rather than
trusting the summarized sessions_history tool output or the gateway log alone.
Actual mechanism: call #1 produced a real, substantive reply and a tool call (a memory
write). The tool ran. Call #2 — the required follow-up turn after any tool call resolves — was
the model, in isolation, judging "nothing more to add" and outputting the literal string
NO_REPLY as that follow-up's entire text. The turn dispatcher uses only the final model
call's text as the turn's reply — it does not concatenate or fall back to an earlier call's
substantive output. So the real, generated reply from call #1 was silently thrown away.
This is not audio-specific and not session-isolation-related at all — it will reproduce on
any turn where the model makes a tool call and then signs off the mandatory follow-up call
with NO_REPLY instead of substantive closing text (or a repeat of the earlier substance).
Reserve NO_REPLY for turns where it is the model's only output for the whole turn (the
documented ambient/group-silence use case) — never as a "nothing further to add" sign-off after
a tool call already produced something worth sending. sessions_history's session:<hash>
labels on entries like these are just internal turn-grouping IDs for the multiple model calls
within one turn, not evidence of a separate isolated agent session — don't read them as such.
Structural mitigation, built and live-verified same day. A model's own commitment to "stop
doing this" is not enough on its own — it recurred twice more within the hour, on a plain
text turn (not audio), confirming the failure is generic to any tool-call-then-final-call turn,
not tied to the original audio trigger. Built a small local plugin (before_agent_finalize
hook — the documented extension point for "inspect the natural final answer and request one
more model pass") that detects the exact bad shape (final text is literally NO_REPLY, an
earlier assistant message in the same turn had substantive text, and a tool call happened
somewhere in between) and returns { action: "revise", retry: { instruction, maxAttempts } }
to force one more pass instead of letting the turn dispatch silently.
v1 caught the pattern but wasn't sufficient: the guard correctly fired
(before_agent_finalize requested revision), but the forced retry pass also ended in
NO_REPLY — because the retry instruction only described the situation and asked the model
to decide again, the same judgment call it had already gotten wrong once in that same turn.
With maxAttempts: 1, one failed retry meant the harness gave up and dispatched silently
anyway — a real second failure on a live production turn, not a hypothetical.
v2 fix, confirmed live: extract the actual substantive text found earlier in the turn and
hand it back to the retry pass verbatim, inside the instruction ("send this exact text");
the retry's job becomes mechanical repetition instead of a second decision. Bumped
maxAttempts 1→2 as cheap defense-in-depth, not the primary fix. Verified two ways before
trusting it: (1) confirmed the deployed file actually contained v2's substantiveTexts/
recoveredText extraction, not v1's plain boolean check; (2) re-ran the exact synthetic repro
that had caught the v1 failure, isolated in a subagent so it wouldn't touch the live
conversation — the retry pass's actual output was the distinctive marker text handed to it,
sent verbatim, stopReason=stop, no silent-drop warning. A real pass, not just "the hook
fired." Treat this as closing the specific tool-call-then-NO_REPLY shape, not as proof no
further NO_REPLY-adjacent failure mode exists — worth watching over further real incidents
rather than treating as permanently settled.
Design lesson for any similar guard: a before_agent_finalize (or similar) revise/retry
mechanism is only as good as the instruction it hands back. Asking the model to re-decide
something it already got wrong once in the same turn has no reason to succeed the second time
either — extracting and replaying the actual prior content, turning the retry into mechanical
repetition rather than a fresh judgment call, is what actually closed the gap here.
Full closure, 2026-08-23: Olu relocated the plugin to a permanent home
(~/.openclaw/workspace/plugins/no-reply-guard/), caught and fixed its own regression during
that move (a full uninstall/reinstall accidentally dropped plugins.entries.no-reply-guard. hooks.allowConversationAccess, silently disabling the hook — caught via typedHooks coming
back empty on inspection, before declaring success), and confirmed a live synthetic test passed
post-restart with the actual recovered marker text delivered verbatim, not another silent
NO_REPLY. Re-verification was briefly blocked by the model-corruption bug below recurring
rapidly (three synthetic-test subagent attempts in close succession each apparently
re-triggering it) — Olu correctly recognized the known-bug signature via its own investigation
and stopped retrying rather than hammering a known-bad window, deferring to a human decision on
whether to force an out-of-cooldown restart. Good instinct on both counts (self-caught
regression via direct state inspection, not blind retry-until-success).
Filed upstream: openclaw/openclaw#128314
— related to, but distinct from, the already-fixed
#93166/#116006
(transcript-isolation fix, merged 2026-07-29, confirmed present in the 2026.7.1-2 install used
here). #116006 fixed the retry pass seeing its own rejected draft in the transcript; the bug
above is that the retry pass can independently produce NO_REPLY again even without seeing
any prior draft, because the natural retry instruction just re-poses the same judgment call.
A version upgrade can silently move a lane onto the exact-match-only silence token, leaking real content that used to be safe
Real incident, 2026-09-04 on dfw, distinct mechanism from the tool-call-then-NO_REPLY bug
above (that one drops a real reply; this one leaks one, sentinel included). Will reported
heartbeat/watcher status messages arriving on Telegram with a literal trailing NO_REPLY —
e.g. "All healthy — heartbeat, watcher, and weekly jobs are enabled...\n\nNO_REPLY" delivered
verbatim, not suppressed.
Root cause, confirmed by diffing the live 2026.8.2 dist against the prior 2026.6.34 build
(pulled from local npm cache, integrity-verified) and independently re-verified line-by-line
against the running server: the 2026.8.2 upgrade changed the default heartbeat prompt
itself, from "reply HEARTBEAT_OK" to "reply ${SILENT_REPLY_TOKEN}" (i.e. NO_REPLY) —
moving heartbeat turns onto the same exact-match-only silence contract used elsewhere, for the
first time. normalizeHeartbeatReply (heartbeat-runner-jhGs3jbv.js) has dedicated
edge/punctuation stripping for HEARTBEAT_OK, built when that was the only heartbeat token —
NO_REPLY never got equivalent treatment when the prompt was switched, so a GLM sign-off habit
(real status content, then NO_REPLY appended as a closer) fails the exact-match silence check,
gets classified as real visible output, and is delivered whole. A sibling delivery path (cron/
isolated, normalizeSilentReplyText in delivery-dispatch-policy-CuyV2VD3.js) has the opposite
bug on the same mixed-text shape: it does detect and strip the trailing token, but then
discards the entire stripped remainder instead of delivering it — silent data loss rather than a
leak. Both are one-line-class fixes (strip and deliver the remainder), neither shipped.
General lesson: an upgrade can silently widen which lanes use a given sentinel/contract without updating every strip/normalize function that assumes the old contract — the failure doesn't show up as a crash or error, just previously-safe content leaking or vanishing on the newly-migrated lane. When a version bump changes default prompts, grep the dist for every strip/normalize function touching the affected token, not just the ones already known.
Diagnostic path that worked: journal (journalctl -u openclaw.service) never logs message
text, only structural events (outbound send ok with a messageId/timestamp, no body) — so
confirming which deliveries leaked required a human forwarding the actual Telegram screenshots
with timestamps, then matching those exact timestamps against journal outbound send
messageIds. Olu's own transcript search initially came back clean ("zero since the upgrade")
because it only scanned conversational session transcripts — heartbeat/cron deliveries don't
persist through that path, a blind spot Olu had already self-flagged. The real diagnosis needed
both sides: Claude Code's root journal access (timing/structure) plus Olu's dist-source access
(the actual code, which Claude Code can't read — lives under /home/openclaw, off-limits per
[[feedback_no_claude_code_home_openclaw_access]]) plus a live npm-cache diff against the prior
release (Olu's own follow-up, not initially requested, that found the real upstream trigger).
Mitigation live: Olu adjusted the heartbeat prompt behavior so it stops appending the sign-off token after real content — confirmed holding via journal cadence (no further leaked deliveries; message spacing after the fix matches live conversational replies, not the ~25-30min heartbeat interval). This is a band-aid, not the structural fix.
Filed upstream: openclaw/openclaw#138229 — includes the refined dist-diff root cause as a follow-up comment. Distinct from and not a duplicate of #128314 (this file, above) or #137024 (an exact-token heartbeat reply getting replaced by an unwanted fallback placeholder — the opposite direction: over-eager stripping causing an unwanted delivery, vs. this bug's no-stripping-at-all).
Restart=on-failure does not cover OpenClaw's own graceful "supervisor restart" exit — the service goes down and stays down
Real outage, 2026-08-23, caused by legitimate OpenClaw behavior, not a misconfiguration on our
side: when OpenClaw's gateway process receives SIGUSR1, it performs [gateway] restart mode: full process restart (supervisor restart) — a deliberate, clean self-exit
(code=exited, status=0/SUCCESS) that assumes an external process supervisor (systemd, pm2,
etc.) will relaunch it immediately. This is a legitimate, documented-sounding self-restart
mechanism (used here when a plugin config change needed a full reload, e.g. after moving a
plugin to a new install location) — not a crash.
The trap: a systemd unit with the seemingly-reasonable Restart=on-failure does not
restart on this exit — on-failure only covers a non-zero exit code or certain fatal signals,
and a clean status=0 exit is explicitly excluded by that policy's own definition. The service
just goes inactive (dead) and stays there indefinitely — no crash loop, no restart attempt,
no alert, nothing in the journal after the clean shutdown lines. Caught here only because a
newly-built watchdog's manual test run happened to overlap with a real SIGUSR1 restart Olu
triggered on its own (relocating a plugin), and a routine post-action status check found the
service dead several minutes after the fact — this could easily have gone unnoticed far longer
in a less actively-monitored moment. Confirmed via systemctl status: Process: ... (code=exited, status=0/SUCCESS), Deactivated successfully, with the unit never re-entering
activating.
Fix: change the unit's Restart= from on-failure to always — the standard policy for
any long-running service that should never intentionally stay down, covering both crashes and
this kind of deliberate clean-exit-expects-relaunch pattern. systemctl daemon-reload is
sufficient to apply it (no service restart needed — the policy only governs what happens on
the next exit). Verified by reproducing the exact failure: sent SIGUSR1 directly to the
live process after the fix and confirmed systemd auto-relaunched it within seconds with no
manual intervention, vs. the original incident where nothing brought it back at all.
If any other OpenClaw deployment shows the same Restart=on-failure in its unit (common
default, looks like the "safe, minimal" choice), check for restart mode: ... (supervisor restart) anywhere in that instance's own history before assuming crash-only coverage is
sufficient — any deployment that installs plugins, changes config needing a full reload, or
otherwise triggers this self-restart path has the same exposure.
A root-owned dispatch script bypasses openclaw.service's own sandboxing entirely — that's a feature, use it deliberately
Confirmed 2026-09-04: openclaw.service's ProtectSystem=strict makes /usr/lib/node_modules
(the npm global install root) read-only to the sandboxed gateway process itself — Olu hit a
literal EROFS trying to upgrade its own runtime, correctly concluded it structurally could not
do this, and was right. But the existing npm-global-install subcommand in
openclaw-dfw-admin-changes.sh (see the dispatch-script pattern above) already lets Olu install
other packages (wrangler is the current allowlisted one) — this works precisely because that
script runs as root via a NOPASSWD sudoers grant, entirely outside openclaw.service's own
unit and its ProtectSystem=strict scope. The sandbox only ever constrains the confined
process's own writes; a separate root-owned script invoked via sudo was never inside that
boundary to begin with.
Don't extend that same generic subcommand to let OpenClaw upgrade itself, though. A
standalone CLI tool like wrangler has no runtime state to break — OpenClaw upgrading its own
running gateway is categorically different (see the auto-update-crash incident above: a bare
version bump left the process running fine until the next restart, which then hit a chain of
undocumented legacy-migration gates with a broken doctor --fix). If this capability is ever
built, it needs its own subcommand with real structure — a pinned target version rather than
"any version," an automatic pre-upgrade config+session backup (Olu already does this by hand;
should be built into the capability, not a manual habit), and probably no auto-restart, so a bad
upgrade doesn't cascade unattended. Treat it as a real design task (this repo's own "propose
before building" convention), not a same-session manifest edit.
A revoked/rotated Telegram token crash-loops the channel with a clear signature — and the fix is often outside the /home/openclaw access boundary
Real incident, 2026-08-29: a BotFather mix-up during an unrelated credential rotation
accidentally revoked @dfwclaw_bot's own live token. Failure signature in journalctl -u openclaw.service is unambiguous and self-diagnosing — the error names the exact fields to
check:
[telegram] [default] Telegram bot token unauthorized for account "default" (getMe returned 401
from Telegram; source: config token). Update channels.telegram.botToken,
channels.telegram.tokenFile, or TELEGRAM_BOT_TOKEN with the current BotFather token.
[telegram] [default] channel exited: ...
[telegram] [default] auto-restart attempt N/10 in <backoff>s
Backoff climbs toward 300s over ~10 attempts, then a [health-monitor] restart resets the
counter and it repeats — a real outage, not a one-off blip, until the token is actually fixed.
On this deployment, TELEGRAM_BOT_TOKEN resolved to /root/.config/openclaw-anthropic.env
(root-owned, wired into the unit via EnvironmentFile= — confirm with systemctl cat openclaw.service | grep EnvironmentFile), alongside the other provider keys — not a plaintext
value inside ~/.openclaw/openclaw.json. This file lives outside /home/openclaw entirely,
so it is not covered by the harness's /home/openclaw access restriction (see
feedback_no_claude_code_home_openclaw_access in the homelab planning repo's memory) — Claude
Code can read field names (cut -d= -f1, no values) and the systemd unit, and can run the
restart + verification, even though it still can't see or write the actual secret value (the
human writes that directly into the file, standard credential-rotation-protocol pattern). Don't
assume every OpenClaw-token problem is blocked by that boundary — check whether the value is
actually env-sourced via a root-owned EnvironmentFile first, since that's a very common pattern
for provider keys per the "Secrets: two valid patterns" section above.
Fix: human edits the TELEGRAM_BOT_TOKEN= line in place (sudo nano <path>, never a shell
command-line argument — keeps it out of bash history), then sudo systemctl restart openclaw.service. OpenClaw handles the rotation gracefully on its own — no extra step
needed for the stale Telegram update offset:
[telegram] [default] starting provider (@dfwclaw_bot)
[telegram] Detected token rotation for account "default" (was <id>, now <id>); discarding stale
update offset <N> and starting fresh.
Verify clean via journalctl -u openclaw.service --since <restart-time> | grep -i telegram —
should show the two lines above and nothing matching unauthorized|401|exited|auto-restart.
Does a release contain a fix? Compare call order in the published dist, not a comment grep (2026-09-12)
The upstream fix for a recurring cron error (Failed to remove unused run continuation … competing work in flight) was identified on main by a source comment, but the bundled dist strips comments.
Grepping two published versions for that comment found it in neither, which proves nothing. What
decided it was the code order inside the post-run finally:
- 2026.8.2 (
dist/isolated-agent-*.js):removeCronRunContinuationSessionIfIdle()L1552, thensessionWorkAdmission.release()L1563. It deletes while its own admission is still held, which is the race. - 2026.9.4 (
dist/isolated-agent-*.mjs): release L1507, then remove L1510. Fixed.
Method, with nothing installed:
T=$(mktemp -d); cd "$T"; npm pack openclaw@<ver> --silent
mkdir <ver> && tar -xzf openclaw-<ver>.tgz -C <ver>
grep -nE "<fnA>\(|<fnB>\(|finally ?\{" <ver>/package/dist/<chunk>
Do this in a temp dir outside the service account's home. The error was log noise only (every run
ok), which is itself a reason to schedule the upgrade calmly rather than rushing it.
OpenClaw has a real background auto-updater (update.auto.enabled) — check it, but verify before blaming it
Confirmed via the installed dist bundle (/usr/lib/node_modules/openclaw/dist/update-startup-*.js,
schema-*.js): OpenClaw ships a genuine background auto-update feature, config key
update.auto.enabled, documented default false. Worth checking (openclaw config get update.auto.enabled, run by the account owner — this is inside the service account's config,
off-limits to Claude Code per the access-boundary note below) on any deployment as a matter of
awareness. But don't assume it's the cause of an unexplained version jump without verifying —
on the incident below, this was the first theory and it was wrong: the live config and all 7
on-disk config backups had the key unset (default false, genuinely off), and the real cause
turned out to be a human running npm install -g openclaw@<version> manually in a real
interactive terminal. The tell that separates the two: check journalctl _COMM=sudo (not just
journalctl -u openclaw.service) for the transition window — a manual update shows up there
with a real TTY=pts/N, distinct from anything Claude Code or an internal auto-updater would do.
A restart can be the moment an already-applied update's migration debt comes due — whether the update was manual or automatic
Real incident, 2026-09-04 (full narrative: project_openclaw_autoupdate_crash_2026_09_04 memory
in the homelab planning repo). OpenClaw was updated from 2026.7.1-2 to 2026.8.2 (manually, by the
operator, per the correction above) — the version bump alone didn't crash anything, because the
running process kept running fine on the old in-memory code. The crash only surfaced ~17
minutes later on the next full process restart (for an unrelated reason), which hit a chain of
three sequential legacy-state migration gates the old version had never needed. Lesson: if
openclaw.service needs restarting for any reason, check openclaw --version against what you
last confirmed first, and be mentally ready for a chain of migration gates instead of a clean
restart — a version change (deliberate or not) may be sitting there unnoticed since the last
restart.
openclaw doctor --fix itself may be completely non-functional on a hand-authored systemd
unit. On this deployment (system-level unit with a custom ExecStart wrapping the binary
through Agent Vault's MITM proxy — see agent-vault-credential-broker skill — not installed via
openclaw gateway install --system), every doctor --fix/--fix --force invocation failed
identically:
Doctor could not enter maintenance. Stop the Gateway through its service owner, then run
openclaw doctor --fix. Error: Gateway service ownership or shutdown could not be verified.
Run openclaw gateway status --deep and stop it through its service owner before retrying.
This persisted even with the Gateway genuinely, repeatedly verified stopped (systemctl is-active = inactive, no matching process), across multiple retries, with and without an
explicit XDG_RUNTIME_DIR=/run/user/<uid> in the invoking environment, and after confirming
(via direct manual systemctl --user/busctl --user calls, bypassing the CLI entirely) that
the D-Bus session bus for the service account was genuinely up and working. Do not assume this
is the well-documented openclaw/openclaw#137503 (missing dbus-user-session package) without
independently verifying — that package can already be installed and the session bus can already
be live, and the generic error persists anyway for a still-unidentified reason. Filed as a fresh
issue: openclaw/openclaw#137888.
The actual fix: bypass doctor --fix entirely and use its narrower, documented sub-commands
one legacy-state class at a time — each of these worked non-interactively where the umbrella
--fix/--repair could not (a documented general pattern, see closed issues #134036 and
#134331 — non-TTY/automation contexts silently skip or deadlock the umbrella migrations):
Legacy workspace setup state requires migration for <workspace>— per openclaw/openclaw#137866/#134331, the startup guard checks THREE possible legacy source markers before it'll proceed, existence-only (it throws on presence, never reads content):<workspace>/openclaw-workspace-state.json<workspace>/.openclaw/workspace-state.json~/.openclaw/workspace-attestations/<sha256-of-resolved-workspace-path>.attested— compute the filename withpython3 -c "import hashlib; print(hashlib.sha256(b'/abs/path/to/workspace').hexdigest())"Back up and remove whichever of the three actually exist. The canonical SQLite state usually already has the correct data — these are safe, redundant files once superseded, not the only copy of anything, unlike the next one.
Legacy session store requires migration: <path>/sessions/sessions.json— this one DOES hold real conversation/session history, so don't just delete it. Use the narrow, safe, additive sub-command instead (preview first, it's read-only):openclaw doctor --session-sqlite dry-run --session-sqlite-agent <agent-id> --json openclaw doctor --session-sqlite import --session-sqlite-agent <agent-id> --jsonimportmoves legacy JSONL files into an archive directory (not deleted) and reportsimportedEntries/importedTranscriptEventscounts — verify these matchlegacyEntriesbefore trusting it fully migrated.Legacy exec approvals exist at <home>/.openclaw/exec-approvals.json— move the file aside (must stay within the service account's own home directory; it can't write to/rooteven undersudo -u), then re-register it:
Verify viasudo -u openclaw mv ~openclaw/.openclaw/exec-approvals.json ~openclaw/.openclaw/exec-approvals.json.moved sudo -u openclaw openclaw approvals set --file ~openclaw/.openclaw/exec-approvals.json.movedopenclaw approvals getthat the full allowlist (target/agent/pattern rows) survived intact before considering this done.
After all three are cleared, expect [gateway] ready in the log but the crash-loop breaker
(tripped during the earlier failed boots) will likely still be suppressing channel auto-start
even on a clean boot — it needs one manual nudge, exactly as its own log line suggests:
openclaw gateway call channels.start --params '{"channel":"telegram"}'
Messages that arrived during the whole outage are queued/retried by the channel provider and delivered once it comes back — confirmed zero message loss across a ~75-minute outage in this incident, no need to worry that queued inbound messages were silently dropped.
Access-boundary note for Claude Code specifically: every one of the above commands (the
cp/mv/openclaw doctor/openclaw approvals invocations) touches /home/<service-account>
directly and must be run by the human operator, not Claude Code — see
feedback_no_claude_code_home_openclaw_access in the homelab planning repo's memory. Claude
Code's role in an incident like this is: systemctl stop/start/restart (fine, doesn't touch the
home directory), reading journalctl output, researching the actual upstream fix commands, and
computing things like the sha256 attestation filename locally — not touching the account's home
directory even read-only (a stray find into it mid-incident, just to look for other candidate
legacy files, was a real overstep in this incident — ask the human to run ls/find themselves
instead, every time, no exceptions for urgency).
Two structurally different run-timeout mechanisms produce near-identical-looking failures — don't conflate them
Confirmed 2026-09-04 on dfw: a user got OpenClaw's generic "Request timed out before a
response was generated. Please try again, or increase agents.defaults.timeoutSeconds in your
config." error on a live chat turn. The journal showed:
[agent/embedded] embedded run timeout: runId=<id> sessionId=<id> timeoutMs=300000
[provider-transport-fetch] [model-fetch] error ... elapsedMs=681 name=TimeoutError message=request timed out
That second line looks like "the model provider is slow/broken" — it isn't. It's the symptom
of the first line: the whole run hit its overall wall-clock budget (agents.defaults. timeoutSeconds, 300s here) and got aborted mid-flight, and the abort surfaces as a spurious
TimeoutError on whatever individual call happened to be in progress at that instant — even if
that specific call had only been running well under a second. Before concluding a model
provider is unhealthy from a TimeoutError log line, check whether every OTHER call in the same
run around that timestamp was fast and successful (grep the surrounding window for
model-fetch] response ... status=200) — if so, it's the run budget, not the provider.
Separately, a cron job's own payload.timeoutSeconds is a distinct, per-job budget (default
much shorter — 120s was the value that bit in the confirmed incident) from the interactive/
heartbeat agents.defaults.timeoutSeconds. A cron job doing real, legitimate multi-tool-call
work (e.g. reading several files, checking multiple sources) can blow straight through a 120s
budget with zero idle time and zero bug — it looks identical in the journal to a stuck/looping
job (lane task error: ... durationMs=120145 error="TimeoutError: cron: job execution timed out (last phase: model-call-started)"), but the fix is right-sizing the job's own timeout to its
real workload, not debugging the trigger-evaluation logic. In the confirmed incident, throwaway
test jobs with trivial payloads "passed" while the real job with substantive work "failed" — the
differential was workload-vs-budget, not anything structural; don't be misled by a passing
control test into assuming the failing job's logic is broken.
Practical takeaway for diagnosing any "OpenClaw timed out" report: identify which of the two
budgets fired (agents.defaults.timeoutSeconds for live/heartbeat runs vs. a specific cron job's
payload.timeoutSeconds) before treating it as a provider-health or trigger-logic bug. Both
manifest as a generic-looking TimeoutError, and neither one means the model call that appears
in the error line was actually slow.
A heartbeat-poll turn can self-extend into an open-ended debugging session with no natural stopping point
Also confirmed 2026-09-04: a routine heartbeat poll found two unrelated cron jobs erroring and turned into a genuine 72-call, multi-hop investigation (source-tracing through the dist bundles, then creating throwaway test cron jobs to empirically narrow the cause) that never converged before hitting the 300s run timeout. Every call did real, distinct work — this is not the same failure shape as a stuck retry loop or call-fanout bug — but a heartbeat turn has no built-in concept of "this has gone deep enough, stop and report back," so a legitimately interesting lead can consume the entire run budget with no final answer produced. If heartbeat runs are expected to stay lightweight, scope that explicitly in the heartbeat prompt (investigate-and-report only, defer genuine deep dives to a dedicated follow-up run) or add a per-run tool-call cap as a hard stop — raising the wall-clock timeout alone doesn't fix this, since a genuinely unbounded investigation will just consume however much budget it's given.
Olu cannot restart itself — a restart it "should have done" always needs root
Confirmed 2026-09-04 when Will observed that Olu had built a requested feature without first
performing an approved gateway restart, which read as the agent ignoring an instruction. It
wasn't: openclaw.service runs with NoNewPrivileges/no-new-privileges sudo and tools.elevated
gated off, so the agent is structurally incapable of restarting its own service. Investigate
before agreeing that an agent skipped a step — the more common explanation is that it never had
the capability.
Corollary for any change requiring a restart: plan for a human or Claude Code to run it, and say so explicitly in the task you hand the agent, rather than leaving it as an implied step it will appear to ignore.
Before restarting, check whether the on-disk bundle is newer than the running process:
sudo systemctl show openclaw -p ExecMainStartTimestamp # when the running process started
sudo stat -c '%y %n' <installed bundle path> # when the code on disk last changed
If the bundle predates the running process, the restart re-runs an identical version with no migration debt — boring and safe. If it postdates it, the restart is the moment an already-applied update's migration debt comes due (see the section above on that). That condition being false is what made the 2026-09-04 crash destructive and a later same-day restart uneventful.
Delivery monitoring belongs out of process — see the agent-delivery-canary skill
OpenClaw's three characteristic failure modes — silent non-delivery (turn dispatch), content leak (sentinel strings reaching the user), and egress silent drops — produce no error line at all, so every journal-scraping monitor is structurally blind to them. The journal logs delivery but never message content:
[telegram] outbound send ok accountId=default chatId=<id> messageId=<n> deliveryKind=text
That asymmetry is what makes the trust split work: a root verifier outside the agent asserts
delivery from the journal, while the agent asserts content, being the only party that can
see the composed text. Full architecture, escalation model, and the failure-path tests in the
agent-delivery-canary skill.
A root-owned system unit needs OPENCLAW_SERVICE_REPAIR_POLICY=external (2026-09-07)
openclaw onboard --install-daemon writes a user unit (~/.config/systemd/user/openclaw-gateway.service);
this project runs a hand-written root-owned system unit (openclaw.service, agent-vault run
wrapper). That difference is the 2026-09-04 doctor --fix failure: doctor's maintenance entry
verifies ownership of the managed unit via systemctl --user, sees a system unit with
LoadState=loaded, seals the service definition and refuses ("Gateway service ownership or
shutdown could not be verified"). openclaw/openclaw#140908 (P0, 2026-09-07) is the same chain on
2026.9.2. The docs' answer for a system unit that owns the lifecycle is
OPENCLAW_SERVICE_REPAIR_POLICY=external: doctor goes diagnostic-only for the service lifecycle
and repairs state against a gateway you stopped. On dfw it is set in the unit and in a root
wrapper, /usr/local/sbin/openclaw-doctor, which is the one sanctioned way to run doctor by hand
(service user, service HOME, the policy exported). Stop the gateway through its owner first for
anything that mutates. Unverified by execution on the maintenance gate itself as of writing
(read-only --lint --json runs fine through it); the staging twin's first rehearsal settles it.
Related, from the same day (Olu's trace of the 2026.8.2 dist): openclaw update classifies a
system unit foreign, stops nothing, and refuses to run from inside the gateway process, so the
safe hop on such a host is stop-via-owner → update --tag <v> --yes --no-restart → start.
Migrations run inside the update's doctor pass, not at boot (boot only has refusal gates), so a
bare restart after a manual npm install -g trips the gates without running the migrations.
update --dry-run does not check migrations; doctor --lint --all --json does. Freeze knobs:
update.auto.enabled: false, update.checkOnStart: false, OPENCLAW_NO_AUTO_UPDATE=1; there is
no update.pin and --tag is not persisted, so the declared version lives in
dfw-ansible's openclaw-gateway-install.yml, which fails on a mismatch by design.
Upgrade 2026.8.2 → 2026.9.6 on dfw (2026-10-07, D-28): what actually bit
- Node first. 9.3+ needs Node ≥24.16 (<25) or ≥26.1. On dfw: switch
/etc/apt/sources.list.d/nodesource.sourcesfromnode_22.xtonode_24.x,apt-get install nodejs(24.21.0). Only the gateway uses Node there. The running gateway keeps the old binary until its restart. 8.2 also runs on Node 24, so an OpenClaw rollback survives. - npm 11 (ships with Node 24) skips install scripts by default. A plain
npm install -g openclaw@X"succeeds" without OpenClaw's bundled-plugins postinstall. Usenpm install -g --allow-scripts=@google/genai,koffi,protobufjs,openclaw openclaw@X, including for a rollback. sudo -u openclaw … openclaw doctorfrom claude-admin's home fails withspawn /usr/bin/node EACCES: the child inherits a cwd openclaw can't enter. Run it ascd / && sudo -u openclaw -H ….- Preview before
doctor --fix. The 9.6 preview wanted to archive TOOLS.md and merge it into AGENTS.md (TOOLS.md is Will's call, never Claude Code's) and to move HEARTBEAT.md into the heartbeat job's scratch (intersects Olu's Lane B row). Neither was run; both went to Will/Olu. The gateway's own startup runs the safe migrations (transcript media, config-audit log → SQLite, Skill Workshop relocation) without them. - Capability consent: non-bundled plugins (admin-changes-gate, no-reply-guard) needed
openclaw plugins enable <id> --accept-capabilitiesbefore restart, or they may not load. admin-changes-gate is Olu's outbound approval gate, so check it is in the "http server listening (N plugins: …)" line. - Version-pinned plugins self-disable: heartbeat-silent-strip and d24-runaway-loop-guard log "version pin mismatch … hook DISABLED (no-op)" until Olu re-validates and bumps the pin. Plan that into the window.
- Plugin API floors: brave 2026.7.1 couldn't be updated because its next release needs plugin API ≥2026.9.7.
- Write
/root/state/openclaw-planned-restart(epoch) before stopping, so the hermes peer watch treats it as planned. - Re-pin version-pinned plugins BEFORE the first restart on the new version. On D-28 they were re-pinned after it, so the first boot ran with heartbeat-silent-strip as a no-op for ~70 min, and two raw heartbeat outputs reached Will's DM (msgs 3428, 3429). It took a third restart to arm them.
- Stuck heartbeat after a restart that interrupts a heartbeat run: the restart-recovery of the interrupted
heartbeat session collided with the next cron run, which then sat with
state.runningAtMsset for over an hour, andcron statusshowednextWakeAtMspinned to it. On 9.6 the heartbeat is a system-owned monitor job:cron disable/enable/editrefuse ("system-owned monitor jobs cannot be edited by cron clients"), and there is no cancel. A gateway restart clears it (the run is recorded as "interrupted by gateway restart"). Afterwards, check thatrunningAtMsgoes null between ticks. - How lane starvation looks on 9.6: no 600 s job kill. Instead the heartbeat logs
skipped: requests-in-flightwhile other turns hold the lane, or runs intoagents.defaults.heartbeat.timeoutSeconds(1800 s on dfw). Long subagents (chat-run timeout ~30–47 min) holding bothmaxConcurrent=2slots also make the 120 s canary time out at "last phase: model-call-started", even while model calls themselves return 200 in 1–3 s. - Paging run history (2026.9.6):
--idneeds the full job UUID; a short prefix returns "Automation not found". Get it fromcron list --json.--limitcaps at 200 ("must be <= 200"), so page with--offset 0/200/400….- Add
--status allto include skips and errors. - Records live under
.entries[], withstatusok/skipped/error,completionStatus,durationMsandrunAtIso. - A-12 was judged this way on 2026-10-09: 293 runs, no 600 s kill, 2 × 1800 s timeouts and 16 skips. That is lane starvation in its 9.6 form, which is a capacity problem, not the old kill.
- Reading runs:
cd / && sudo -u openclaw -H openclaw cron runs --id <job> --limit N --json | jq '[.. | objects | select(has("runAtIso"))] | sort_by(.runAtIso)'(the list comes oldest-last). Don't grep the journal for "timed out": the agent's own report frames quote old errors into it.
Stopping or retiring an OpenClaw agent: disable the restarters first (2026-10-17)
A fleet that babysits OpenClaw will fight a plain systemctl disable --now openclaw. Order that worked on dfw:
- Write the planned-restart marker (epoch into
/root/state/openclaw-planned-restart) so peer/health watchers stand down. disable --nowevery timer that restarts it (corruption watchdog, conditional memory restart) — a watchdog'ssystemctl restartstarts a disabled unit.disable --nowevery timer that alerts on it (health check, provider-error watch, model-drift check), and the peer host's watch if its only subject is this agent. Keep watchers whose subject is the other agent. Include the delivery canary verifier (dfw-canary-verify.timer): it reads onlyjournalctl -u openclaw, so a stopped agent looks like "cannot see the journal" and it pages every 30 min (missed 10-17, 136 alerts by 10-20). Inventory by grepping for the unit, not by timer name:grep -lE 'openclaw\.service|-u openclaw' /usr/local/bin/* /usr/local/sbin/*, then check which of those have a live timer.- Stop
openclaw+openclaw-admin-server. SIGTERM exits 143, so the unit landsfailed:systemctl reset-failed openclawor the fleet failed-unit check pages. - Off-host: pause dispatch/brief scripts and remove any sweep check that expects the dashboard Serve to answer; dry-run both.
- Record that these are hand changes: the Ansible role re-enables them until the role is removed.
Leftover systemd --user/dbus-daemon processes for the service user are the lingering user session, not the agent. Durable removal (vault/keys, egress rules, exchange, fleet access, data archive) is a separate step.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 464k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 464k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 464k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |