Skip to content

notque/vexjoy-agent / skills/infrastructure/deploy

deploy

Flagged

Deploy and infrastructure: web deploy, cron, package install, shell configuration.

references/cleanup-verification.md

View on GitHub
---
name: cleanup-verification
description: Kill-and-check — never trust that a destructive operation succeeded, re-query state.
---

# Cleanup Verification

The core principle: **after any destructive operation on a process, re-query state to verify it actually worked.** `kill` returning 0 means the signal was delivered, not that the process is gone. `rm -f path` returning 0 means the unlink syscall succeeded; it does not tell you the process holding that file as open-fd has stopped writing.

The pattern name is **kill-and-check, not kill-and-assume.**

## Why `kill` exit status is insufficient

`kill` returns:
- `0` — signal was delivered (or was signal 0 and the target exists)
- `1` — permission denied, or target does not exist
- Other — error parsing arguments

What `kill` never tells you:
- Whether the target handled the signal
- Whether the target has finished shutting down
- Whether the target is still holding resources (ports, files, locks)

A process that ignores SIGTERM receives the signal, `kill` exits 0, and the process keeps running forever. A process that handles SIGTERM starts shutdown and may need seconds to finish. Either way, you cannot assume "dead" from `kill`'s exit code.

## The liveness check: `kill -0`

`kill -0 "$PID"` sends signal 0, which the kernel treats as "test whether the PID is valid and we could signal it." No actual signal is delivered.

```bash
if kill -0 "$PID" 2>/dev/null; then
  echo "still alive"
else
  echo "dead or unreachable"
fi
```

Caveats:
- Returns 1 when the PID belongs to another user (permission error) even if alive. Run as the same user or as root.
- Does not distinguish "dead" from "PID recycled by another process." Always verify cmdline or resource ownership when PID recycling matters.

## The standard kill-and-check loop

```bash
terminate_and_verify() {
  local pid=$1
  local grace=${2:-5}   # seconds to wait for graceful exit
  local resource=${3:-} # optional: port or file to re-verify

  # 1. Polite shutdown
  kill -TERM "$pid" 2>/dev/null || true

  # 2. Wait up to grace seconds for the process to exit
  for _ in $(seq 1 $((grace * 10))); do
    kill -0 "$pid" 2>/dev/null || break
    sleep 0.1
  done

  # 3. Escalate if still alive
  if kill -0 "$pid" 2>/dev/null; then
    kill -KILL "$pid" 2>/dev/null || true
    sleep 0.2
  fi

  # 4. Final liveness check
  if kill -0 "$pid" 2>/dev/null; then
    echo "ERROR: PID $pid still alive after SIGKILL" >&2
    return 1
  fi

  # 5. Resource check (more important than PID check)
  if [[ -n "$resource" ]]; then
    case "$resource" in
      port:*)
        local port=${resource#port:}
        if ss -tln 2>/dev/null | grep -q ":${port}\b"; then
          echo "ERROR: port $port still bound after kill" >&2
          return 1
        fi
        ;;
      file:*)
        local f=${resource#file:}
        if lsof -t "$f" 2>/dev/null | read -r; then
          echo "ERROR: file $f still held by some process" >&2
          return 1
        fi
        ;;
    esac
  fi

  return 0
}

# Usage
terminate_and_verify "$PID" 5 port:8080
```

## Why resource verification beats PID verification

A process can exit while its child continues to hold the resource you cared about. A process can double-fork, leaving `$PID` dead but a grandchild alive and still listening on the port. A process can crash and systemd can restart it under a new PID before your verification runs.

Your user cares whether port 8080 is free, not whether a specific PID is gone. Verify the resource.

```bash
# Bad: only checks the one PID we knew about
kill "$PID"
sleep 1
kill -0 "$PID" 2>/dev/null && echo "still alive"

# Good: checks the resource the user cares about
kill "$PID"
sleep 1
if ss -tln | grep -q ':8080\b'; then
  echo "ERROR: port 8080 still bound"
  # escalate: find whatever is holding it now and kill that
  NEW_PID=$(ss -tlnp | awk '/:8080\b/ {print $NF}' | grep -oP 'pid=\K[0-9]+' | head -1)
  kill -KILL "$NEW_PID"
fi
```

## Common resources to verify

| Resource | Verification |
|----------|--------------|
| TCP port | `ss -tln \| grep -q ':PORT\b'` or `lsof -i TCP:PORT` |
| UDP port | `ss -uln \| grep -q ':PORT\b'` |
| Unix socket | `[[ -S /path/to.sock ]]` (path exists) or `lsof /path/to.sock` |
| File lock (flock) | `flock -n /path/to.lockfile -c true` — success = lock is free |
| PID file | `[[ -f /var/run/app.pid ]] && kill -0 "$(<file)"` |
| Log file writer | `lsof -t /var/log/app.log` |
| Mount point | `findmnt /path` |
| Database session | Application-specific query |

## Cleanup on script exit

Combine EXIT trap with kill-and-check:

```bash
#!/usr/bin/env bash
set -euo pipefail

PIDS=()

start_worker() {
  "$@" &
  PIDS+=($!)
}

cleanup() {
  local rc=$?

  for pid in "${PIDS[@]}"; do
    kill -TERM "$pid" 2>/dev/null || true
  done

  # Grace period
  local deadline=$((SECONDS + 5))
  for pid in "${PIDS[@]}"; do
    while kill -0 "$pid" 2>/dev/null && (( SECONDS < deadline )); do
      sleep 0.1
    done
  done

  # Escalate survivors
  for pid in "${PIDS[@]}"; do
    kill -KILL "$pid" 2>/dev/null || true
  done

  exit "$rc"
}

trap cleanup EXIT

start_worker long_task_a
start_worker long_task_b

wait
```

## The "it died but the port is still held" case

This happens when the target process had a child that it failed to kill on its own way out. The parent's PID is gone; a grandchild owns the socket.

```bash
kill -TERM "$PARENT_PID"
# Parent exits, child survives, port stays bound.
```

Recovery: after the first kill, re-resolve the resource owner and kill whatever is holding it now.

```bash
kill_everything_holding_port() {
  local port=$1
  local max_iter=5

  for _ in $(seq 1 $max_iter); do
    local pids
    pids=$(ss -tlnp 2>/dev/null | awk -v p=":${port}" '$0 ~ p' \
           | grep -oP 'pid=\K[0-9]+')
    [[ -z "$pids" ]] && return 0

    while read -r pid; do
      [[ -n "$pid" ]] || continue
      kill -TERM "$pid" 2>/dev/null || true
    done <<< "$pids"

    sleep 0.5

    # Check if port is free now
    ss -tln | grep -q ":${port}\b" || return 0
  done

  # Still bound; escalate to SIGKILL on everything
  ss -tlnp 2>/dev/null | awk -v p=":${port}" '$0 ~ p' \
    | grep -oP 'pid=\K[0-9]+' \
    | while read -r pid; do kill -KILL "$pid" 2>/dev/null || true; done

  sleep 0.2
  ss -tln | grep -q ":${port}\b" && return 1 || return 0
}
```

## Verification in tests

If your test kills a server and then starts a new one, verify the port is free before starting. Otherwise the new server fails with "address already in use" and the test flakes.

```bash
# Teardown
kill -TERM "$SERVER_PID" 2>/dev/null || true
for _ in $(seq 1 30); do
  ss -tln | grep -q ':8080\b' || break
  sleep 0.1
done
ss -tln | grep -q ':8080\b' && { echo "FATAL: teardown incomplete"; exit 1; }

# Now safe to restart
start_server
```

<!-- no-pair-required: This section is a pointer index into preferred-patterns.md; each referenced AP has its own paired "Do instead" block in that file. The body of THIS reference file (kill-and-check loop, resource verification, cleanup-on-exit) IS the "do instead" content for the referenced failure modes. -->

## Failure patterns

- `kill "$PID" && echo done` — `kill` returning 0 is not "done"; see `preferred-patterns.md` AP-4
- `kill -9` as the default — removes the process's chance to clean up; see AP-7
- `sleep 5; echo "should be dead now"` — races; poll the liveness check instead

## Cross-references

- For PID capture before kill: `pid-resolution.md`
- For signals and traps: `signals-and-traps.md`
- For failure modes: `preferred-patterns.md` AP-4, AP-7
- For polling/backoff generalities: `skills/process/process/SKILL.md` (Wait mode)
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall