Skip to content

notque/vexjoy-agent / skills/infrastructure/deploy

deploy

Flagged

Deploy and infrastructure: web deploy, cron, package install, shell configuration.

references/preferred-patterns.md

View on GitHub
---
name: failure modes
description: Concrete shell-process failure modes with detection commands and paired fixes.
---

<!-- no-pair-required: this H1 is the file title; each AP-N section below contains its own "Do instead" block. -->

# Shell Process Patterns to Detect and Fix

Each entry pairs a detection command (runnable against a repo) with the broken pattern and a correct replacement. The detection commands are `rg`-first because ripgrep is available in the toolkit; fall back to `grep -rn` when `rg` is absent.

When you land here, you are probably reviewing a bash script or about to write one. Check your code against each entry below before shipping.

---

## AP-1: `nohup cmd &` captures the wrapper PID, not the child

### Symptom
You start a background daemon with `nohup`, capture `$!`, later run `kill $!`, and the service is still listening on its port. Your script reports success but the process is still alive.

### Why it happens
`nohup` is itself a wrapper process. When you write:

```bash
# local-only test server — bind 127.0.0.1, never on a public VPS
nohup python3 -m http.server 8080 --bind 127.0.0.1 > log 2>&1 &
PID=$!
```

This `http.server` is a local-only test server: it binds `127.0.0.1` so it stays on the loopback interface. For an internet-facing site, route through nginx/Caddy/Apache instead — see the `public-web-deploy` skill. `--bind 0.0.0.0` would expose it on every interface (the public internet on a VPS), which is the wrong tool for a public service.

`$!` holds the PID of the `nohup` wrapper, not the Python interpreter `nohup` forked. On many systems `nohup` exec-replaces itself with the child, so they end up identical; on others (including recent glibc/coreutils combinations) they do not. You cannot rely on them being the same.

Concrete failure (2026-04-21): a Python `http.server` was started with `--bind 0.0.0.0`, binding every interface on a public VPS and exposing the working directory to the internet — the wrong tool for a public site (use the `public-web-deploy` skill). The same run also hit the PID bug: the captured `$!` was PID 1052107 (the `nohup` wrapper), the real listening process was 1052108, `kill 1052107` returned success, and port 8080 stayed `LISTEN`. The server was only killed after `ss -tlnp` was re-queried to find the real PID.

### Detection

```bash
# Find the pattern: nohup followed by & with $! capture
rg -n 'nohup.*&\s*$' --type sh --type bash
rg -n '^\s*PID=\$!' -A 0 --type sh --type bash

# Broader: any `&` + `$!` capture for binaries known to fork
rg -nU '^\s*nohup[^&]*&\s*\n\s*[A-Z_]+=\$!' --type sh --type bash
```

Any hit requires manual inspection: if the backgrounded command is a shell builtin or simple exec it may be safe; if it is anything that daemonizes, forks, or goes through `nohup`/`setsid`, the captured PID is suspect.

### Do instead

Capture the PID by querying the resource the process owns, not by trusting `$!`.

```bash
# Pattern: start, wait for the resource, resolve PID via ss/lsof/pgrep
nohup python3 -m http.server 8080 > log 2>&1 &

# Wait for port to bind
for _ in $(seq 1 30); do
  ss -tln | grep -q ':8080\b' && break
  sleep 0.1
done

# Resolve the real PID from the port owner
PID=$(ss -tlnp | awk '/:8080\b/ {print $NF}' | grep -oP 'pid=\K[0-9]+' | head -1)

if [[ -z "$PID" ]]; then
  echo "ERROR: server did not bind port 8080" >&2
  exit 1
fi

echo "real PID=$PID"
```

For the kill step, verify the port is free afterward, not just that `kill` returned 0. See `cleanup-verification.md`.

Alternative pattern when you really need `$!`: use `exec` inside a subshell to replace the wrapper with the target binary so `$!` points to the real child.

```bash
( exec python3 -m http.server 8080 > log 2>&1 ) &
PID=$!  # This is now the python process itself, not a wrapper
```

But `ss`/`lsof` resource-owner resolution remains more robust because it survives any daemonization pattern.

See `pid-resolution.md` for the full PID capture recipe catalog.

---

## AP-2: `set -e` silently bypassed by `|| true` and friends

### Symptom
A script has `set -e` at the top, a command in the middle fails, and the script continues anyway. Errors are swallowed; the script exits 0 despite real failures.

### Why it happens
`set -e` exits on the first command that returns non-zero. But `set -e` is disabled for any command that is part of:
- A logical operator chain: `cmd || fallback`, `cmd && next`
- A condition in `if`, `while`, `until`
- A pipeline's non-final members when `pipefail` is not set
- A function called in any of the above contexts

The `|| true` idiom is the most common offender. Developers add it to suppress a single expected failure, then the suppression spreads to wrap unrelated logic.

```bash
set -e
rm -rf /tmp/workdir || true  # Intended: tolerate missing dir
# ...100 lines later...
./deploy.sh || true           # Swallowed: deploy fails silently
```

Even worse, functions called under `||` disable `set -e` for their entire body:

```bash
set -e

deploy() {
  build_artifact    # Fails. set -e is DISABLED here because deploy was called under ||.
  upload_artifact   # Runs anyway. Corrupt artifact uploaded.
  restart_service   # Runs anyway. Service restarted with broken artifact.
  return 0
}

deploy || echo "deploy failed"  # Prints nothing. deploy returned 0.
```

### Detection

```bash
# Find `|| true` and `|| :` idioms
rg -n '\|\|\s*(true|:)\b' --type sh --type bash

# Find pipelines without `set -o pipefail`
rg -l 'set -e' --type sh --type bash | while read -r f; do
  if ! rg -q 'set -o pipefail|set -[a-zA-Z]*o[a-zA-Z]*' "$f"; then
    echo "$f: set -e without pipefail"
  fi
done

# Find function calls under || that mask set -e semantics
rg -n '^\s*[a-zA-Z_][a-zA-Z0-9_]*\s*(\(\))?[^=]*\|\|' --type sh --type bash
```

### Do instead

**Always use `set -euo pipefail` together.** Each flag covers a different class of bug:

```bash
#!/usr/bin/env bash
set -euo pipefail

# -e: exit on first unchecked failure
# -u: exit on unset variable reference
# -o pipefail: pipeline fails if any stage fails, not just the last
```

**Handle expected failures explicitly, with scoped checks, not blanket `|| true`.**

```bash
# Wrong
rm -rf /tmp/workdir || true

# Right: test the precondition explicitly
if [[ -d /tmp/workdir ]]; then
  rm -rf /tmp/workdir
fi

# Also right: scope the tolerance to the one command
rm -rf /tmp/workdir 2>/dev/null || [[ ! -d /tmp/workdir ]]
```

**Check exit status inside `if`, not with `||` chains that cascade:**

```bash
# Wrong
build || notify_failure
deploy  # runs even if build failed

# Right
if ! build; then
  notify_failure
  exit 1
fi
deploy
```

**When `|| true` is genuinely needed (e.g. tolerating SIGPIPE from `head`), comment the reason:**

```bash
# head closes stdin on line 10; producer gets SIGPIPE. Tolerated.
producer | head -10 || true
```

---

## AP-3: Trap handler loses the original exit code

### Symptom
Your script has a `trap cleanup EXIT`; when the script fails, `cleanup` runs but the final exit code is the cleanup function's exit code, not the original failure's.

### Detection

```bash
# Traps that run non-trivial logic without preserving $?
rg -nU 'trap\s+[^ ]+\s+EXIT' --type sh --type bash -A 3
```

### Do instead

Capture `$?` as the first line of the trap handler, before any other command can overwrite it.

```bash
cleanup() {
  local rc=$?
  rm -rf "$TMPDIR"
  exit "$rc"
}
trap cleanup EXIT
```

See `signals-and-traps.md` for the full ordering discussion.

---

## AP-4: Kill without verification

### Symptom
`kill "$PID"` returns 0, script proceeds assuming the process is dead, but the process ignored SIGTERM or is still shutting down.

### Detection

```bash
# kill with no follow-up verification
rg -nU 'kill\s+(-[0-9]+\s+)?\$?[A-Z_]+\s*\n' --type sh --type bash -A 2
```

### Do instead

Always re-query state after a destructive action. `kill -0 "$PID"` is the cheapest liveness check.

```bash
kill -TERM "$PID"

# Give it up to 5s to shut down cleanly
for _ in $(seq 1 50); do
  kill -0 "$PID" 2>/dev/null || break
  sleep 0.1
done

# Escalate if still alive
if kill -0 "$PID" 2>/dev/null; then
  kill -KILL "$PID"
  sleep 0.2
fi

# Verify the resource, not just the PID (process could have double-forked)
if ss -tln | grep -q ':8080\b'; then
  echo "ERROR: port 8080 still bound after kill" >&2
  exit 1
fi
```

See `cleanup-verification.md` for patterns that survive double-forking and orphan children.

---

## AP-5: `wait` with no PID kills the script's error handling

### Symptom
`wait` without arguments waits for all background jobs and always returns 0 even if one failed. Subsequent `set -e` logic thinks everything succeeded.

### Detection

```bash
rg -n '^\s*wait\s*

 --type sh --type bash
```

### Do instead

Wait on specific PIDs and check each exit code.

```bash
cmd_a & PID_A=$!
cmd_b & PID_B=$!

wait "$PID_A"
RC_A=$?
wait "$PID_B"
RC_B=$?

if (( RC_A != 0 || RC_B != 0 )); then
  echo "ERROR: one or more background jobs failed (a=$RC_A b=$RC_B)" >&2
  exit 1
fi
```

---

## AP-6: Race between background start and resource readiness

### Symptom
You start a server in the background, then immediately run a client against it. The client fails with "connection refused" because the server has not yet bound the port. Adding `sleep 1` makes it work on your laptop but flakes on CI.

### Detection

```bash
# sleep immediately after background launch is almost always a race hack
rg -nU '&\s*\n\s*sleep\s+[0-9.]+' --type sh --type bash
```

### Do instead

Poll for the resource, not the clock. See the `condition-based-waiting` skill for the full polling/backoff catalog; for simple port-ready checks:

```bash
start_server &

# Poll up to 10s for port to bind
for _ in $(seq 1 100); do
  ss -tln | grep -q ':8080\b' && break
  sleep 0.1
done

ss -tln | grep -q ':8080\b' || { echo "ERROR: server failed to bind"; exit 1; }
```

---

## AP-7: `SIGKILL` as the default

### Symptom
Scripts start with `kill -9` instead of `kill -TERM`. Processes have no chance to flush buffers, close files, or release locks. Corruption follows.

### Detection

```bash
rg -n 'kill\s+-9\b|kill\s+-(KILL|SIGKILL)\b' --type sh --type bash
```

### Do instead

Default to SIGTERM. Escalate to SIGKILL only after a grace period. See AP-4 above for the escalation pattern.

The only valid first-reach SIGKILL is when you have prior evidence the process ignores SIGTERM (document this in a comment), or when the process is known-unresponsive and holding a critical lock.

---

## Cross-references

- For the PID resolution recipes (AP-1): `pid-resolution.md`
- For the full signals and traps discussion (AP-3): `signals-and-traps.md`
- For verification patterns (AP-4, AP-7): `cleanup-verification.md`
- For starting-process idioms (AP-6): `starting-processes.md`
- For polling/backoff (AP-6): `skills/process/process/SKILL.md` (Wait mode)
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall