notque/vexjoy-agent / skills/infrastructure/deploy
deploy
FlaggedDeploy and infrastructure: web deploy, cron, package install, shell configuration.
references/cleanup-verification.md
View on GitHub---
name: cleanup-verification
description: Kill-and-check — never trust that a destructive operation succeeded, re-query state.
---
# Cleanup Verification
The core principle: **after any destructive operation on a process, re-query state to verify it actually worked.** `kill` returning 0 means the signal was delivered, not that the process is gone. `rm -f path` returning 0 means the unlink syscall succeeded; it does not tell you the process holding that file as open-fd has stopped writing.
The pattern name is **kill-and-check, not kill-and-assume.**
## Why `kill` exit status is insufficient
`kill` returns:
- `0` — signal was delivered (or was signal 0 and the target exists)
- `1` — permission denied, or target does not exist
- Other — error parsing arguments
What `kill` never tells you:
- Whether the target handled the signal
- Whether the target has finished shutting down
- Whether the target is still holding resources (ports, files, locks)
A process that ignores SIGTERM receives the signal, `kill` exits 0, and the process keeps running forever. A process that handles SIGTERM starts shutdown and may need seconds to finish. Either way, you cannot assume "dead" from `kill`'s exit code.
## The liveness check: `kill -0`
`kill -0 "$PID"` sends signal 0, which the kernel treats as "test whether the PID is valid and we could signal it." No actual signal is delivered.
```bash
if kill -0 "$PID" 2>/dev/null; then
echo "still alive"
else
echo "dead or unreachable"
fi
```
Caveats:
- Returns 1 when the PID belongs to another user (permission error) even if alive. Run as the same user or as root.
- Does not distinguish "dead" from "PID recycled by another process." Always verify cmdline or resource ownership when PID recycling matters.
## The standard kill-and-check loop
```bash
terminate_and_verify() {
local pid=$1
local grace=${2:-5} # seconds to wait for graceful exit
local resource=${3:-} # optional: port or file to re-verify
# 1. Polite shutdown
kill -TERM "$pid" 2>/dev/null || true
# 2. Wait up to grace seconds for the process to exit
for _ in $(seq 1 $((grace * 10))); do
kill -0 "$pid" 2>/dev/null || break
sleep 0.1
done
# 3. Escalate if still alive
if kill -0 "$pid" 2>/dev/null; then
kill -KILL "$pid" 2>/dev/null || true
sleep 0.2
fi
# 4. Final liveness check
if kill -0 "$pid" 2>/dev/null; then
echo "ERROR: PID $pid still alive after SIGKILL" >&2
return 1
fi
# 5. Resource check (more important than PID check)
if [[ -n "$resource" ]]; then
case "$resource" in
port:*)
local port=${resource#port:}
if ss -tln 2>/dev/null | grep -q ":${port}\b"; then
echo "ERROR: port $port still bound after kill" >&2
return 1
fi
;;
file:*)
local f=${resource#file:}
if lsof -t "$f" 2>/dev/null | read -r; then
echo "ERROR: file $f still held by some process" >&2
return 1
fi
;;
esac
fi
return 0
}
# Usage
terminate_and_verify "$PID" 5 port:8080
```
## Why resource verification beats PID verification
A process can exit while its child continues to hold the resource you cared about. A process can double-fork, leaving `$PID` dead but a grandchild alive and still listening on the port. A process can crash and systemd can restart it under a new PID before your verification runs.
Your user cares whether port 8080 is free, not whether a specific PID is gone. Verify the resource.
```bash
# Bad: only checks the one PID we knew about
kill "$PID"
sleep 1
kill -0 "$PID" 2>/dev/null && echo "still alive"
# Good: checks the resource the user cares about
kill "$PID"
sleep 1
if ss -tln | grep -q ':8080\b'; then
echo "ERROR: port 8080 still bound"
# escalate: find whatever is holding it now and kill that
NEW_PID=$(ss -tlnp | awk '/:8080\b/ {print $NF}' | grep -oP 'pid=\K[0-9]+' | head -1)
kill -KILL "$NEW_PID"
fi
```
## Common resources to verify
| Resource | Verification |
|----------|--------------|
| TCP port | `ss -tln \| grep -q ':PORT\b'` or `lsof -i TCP:PORT` |
| UDP port | `ss -uln \| grep -q ':PORT\b'` |
| Unix socket | `[[ -S /path/to.sock ]]` (path exists) or `lsof /path/to.sock` |
| File lock (flock) | `flock -n /path/to.lockfile -c true` — success = lock is free |
| PID file | `[[ -f /var/run/app.pid ]] && kill -0 "$(<file)"` |
| Log file writer | `lsof -t /var/log/app.log` |
| Mount point | `findmnt /path` |
| Database session | Application-specific query |
## Cleanup on script exit
Combine EXIT trap with kill-and-check:
```bash
#!/usr/bin/env bash
set -euo pipefail
PIDS=()
start_worker() {
"$@" &
PIDS+=($!)
}
cleanup() {
local rc=$?
for pid in "${PIDS[@]}"; do
kill -TERM "$pid" 2>/dev/null || true
done
# Grace period
local deadline=$((SECONDS + 5))
for pid in "${PIDS[@]}"; do
while kill -0 "$pid" 2>/dev/null && (( SECONDS < deadline )); do
sleep 0.1
done
done
# Escalate survivors
for pid in "${PIDS[@]}"; do
kill -KILL "$pid" 2>/dev/null || true
done
exit "$rc"
}
trap cleanup EXIT
start_worker long_task_a
start_worker long_task_b
wait
```
## The "it died but the port is still held" case
This happens when the target process had a child that it failed to kill on its own way out. The parent's PID is gone; a grandchild owns the socket.
```bash
kill -TERM "$PARENT_PID"
# Parent exits, child survives, port stays bound.
```
Recovery: after the first kill, re-resolve the resource owner and kill whatever is holding it now.
```bash
kill_everything_holding_port() {
local port=$1
local max_iter=5
for _ in $(seq 1 $max_iter); do
local pids
pids=$(ss -tlnp 2>/dev/null | awk -v p=":${port}" '$0 ~ p' \
| grep -oP 'pid=\K[0-9]+')
[[ -z "$pids" ]] && return 0
while read -r pid; do
[[ -n "$pid" ]] || continue
kill -TERM "$pid" 2>/dev/null || true
done <<< "$pids"
sleep 0.5
# Check if port is free now
ss -tln | grep -q ":${port}\b" || return 0
done
# Still bound; escalate to SIGKILL on everything
ss -tlnp 2>/dev/null | awk -v p=":${port}" '$0 ~ p' \
| grep -oP 'pid=\K[0-9]+' \
| while read -r pid; do kill -KILL "$pid" 2>/dev/null || true; done
sleep 0.2
ss -tln | grep -q ":${port}\b" && return 1 || return 0
}
```
## Verification in tests
If your test kills a server and then starts a new one, verify the port is free before starting. Otherwise the new server fails with "address already in use" and the test flakes.
```bash
# Teardown
kill -TERM "$SERVER_PID" 2>/dev/null || true
for _ in $(seq 1 30); do
ss -tln | grep -q ':8080\b' || break
sleep 0.1
done
ss -tln | grep -q ':8080\b' && { echo "FATAL: teardown incomplete"; exit 1; }
# Now safe to restart
start_server
```
<!-- no-pair-required: This section is a pointer index into preferred-patterns.md; each referenced AP has its own paired "Do instead" block in that file. The body of THIS reference file (kill-and-check loop, resource verification, cleanup-on-exit) IS the "do instead" content for the referenced failure modes. -->
## Failure patterns
- `kill "$PID" && echo done` — `kill` returning 0 is not "done"; see `preferred-patterns.md` AP-4
- `kill -9` as the default — removes the process's chance to clean up; see AP-7
- `sleep 5; echo "should be dead now"` — races; poll the liveness check instead
## Cross-references
- For PID capture before kill: `pid-resolution.md`
- For signals and traps: `signals-and-traps.md`
- For failure modes: `preferred-patterns.md` AP-4, AP-7
- For polling/backoff generalities: `skills/process/process/SKILL.md` (Wait mode)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |