Skip to content

notque/vexjoy-agent / skills/infrastructure/deploy

deploy

Flagged

Deploy and infrastructure: web deploy, cron, package install, shell configuration.

references/logging-and-rotation.md

View on GitHub
# Logging and Log Rotation Reference

> **Scope**: Cron script logging patterns — timestamps, structured output, log rotation, stderr/stdout routing
> **Version range**: bash 4.0+, GNU coreutils date (all versions), logrotate 3.x
> **Generated**: 2026-04-16

---

## Overview

Cron jobs run headless — their output goes to the cron daemon (often emailed to root, often discarded). Without explicit logging to files, there is no record of what happened, when it ran, or why it failed. With logging but no rotation, log files grow unbounded until they fill the disk. The two most common cron logging failures: no timestamps (impossible to correlate with events), and no rotation (disk fills silently over weeks).

---

## Pattern Table

| Pattern | Use When | Avoid When |
|---------|----------|------------|
| `exec >> "$LOG" 2>&1` | Logging all output to file (headless cron) | Need terminal output too |
| `exec > >(tee -a "$LOG") 2>&1` | Need output in log file AND terminal | Logging only to file |
| `date '+%Y-%m-%d %H:%M:%S'` | ISO timestamps in log lines | Using `$(date)` alone (locale-dependent) |
| `find logs -mtime +N -delete` | Simple rotation without logrotate | High-volume logs needing compression |
| `logrotate -s STATE CONFIG` | Production services, compressed archives needed | Simple scripts (overkill) |
| `LOG="logs/script_$(date +%Y%m%d).log"` | Daily log files (self-rotating by name) | Logs needed across multiple days in one file |

---

## Correct Patterns

### Full logging setup (recommended default)

```bash
#!/bin/bash
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
LOG_DIR="$SCRIPT_DIR/logs"
LOG_FILE="$LOG_DIR/$(basename "$0" .sh)_$(date +%Y%m%d).log"

mkdir -p "$LOG_DIR"
exec >> "$LOG_FILE" 2>&1   # redirect all stdout+stderr to log file

echo "=== $(date '+%Y-%m-%d %H:%M:%S') Starting $(basename "$0") ==="
```

**Why**: `exec >> "$LOG_FILE" 2>&1` redirects all subsequent stdout and stderr to the log file. The daily filename (`_20260416.log`) provides natural rotation — yesterday's log is a separate file. The ISO timestamp in the `echo` line makes log correlation to events straightforward.

---

### Timestamped log function

```bash
log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') [$(basename "$0")] $*"
}

log "Starting backup"
log "ERROR: rsync failed" >&2
```

**Why**: A `log()` function ensures every log line has a consistent timestamp and script name prefix. Easier to grep in a combined log file when multiple cron jobs share a log directory.

---

### Log rotation with find

```bash
# At the end of the script, after work is done
LOG_RETENTION_DAYS=30
find "$LOG_DIR" -name "*.log" -mtime +"$LOG_RETENTION_DAYS" -delete
log "Cleaned logs older than $LOG_RETENTION_DAYS days"
```

**Why**: `find -mtime +N -delete` removes files older than N days in one command. Run at the end of the script so rotation happens after logging the current run. Use a variable for retention days — magic numbers in scripts are hard to maintain.

---

### Separate stdout and stderr log files

```bash
LOG="logs/job_$(date +%Y%m%d).log"
ERR="logs/job_$(date +%Y%m%d).err"
exec >> "$LOG" 2>> "$ERR"
```

**Why**: Separating stdout and stderr makes error detection trivial: `[ -s "$ERR" ]` (true if file is non-empty) can drive failure notification. Useful when success output is verbose and you only want to alert on error output.

---

<!-- no-pair-required: section heading; individual failure modes below carry Do-instead blocks -->
## Pattern Catalog

### Redirect Output to Log Files

**Detection**:
```bash
grep -rL '>> .*log\|tee.*log\|LOG=' --include="*.sh" scripts/ cron/ jobs/
rg -L '>> .*\.log|tee .*\.log|LOG=' --type sh
```

**Signal**:
```bash
#!/bin/bash
set -e
cd /var/data
python3 process.py
rsync -avz output/ backup/
```

**Why this matters**: All output goes to cron's mail system (usually `nobody@localhost`, usually discarded). When the job fails, there is no record of what was running, what the error was, or when the last successful run occurred. Post-incident investigation is impossible.

**Do instead:** Add `exec >> "$LOG_FILE" 2>&1` near the top, after defining `LOG_FILE`.

---

### Use ISO Timestamps in All Log Output

**Detection**:
```bash
  # Scripts using bare $(date) without format string
grep -rn '\$(date)[^+]' --include="*.sh" scripts/ cron/
  # Scripts with echo/printf but no date at all
grep -rl "echo\|printf" --include="*.sh" scripts/ | xargs grep -L "date"
```

**Signal**:
```bash
echo "Starting job"   # no timestamp at all

  # or locale-dependent:
echo "$(date): Starting"  # Thu Apr 16 20:07:00 UTC 2026 (hard to sort/grep)
```

**Why this matters**: Without timestamps, log lines from different runs blend together. Locale-dependent `date` output sorts lexicographically wrong and is hard to parse programmatically. In a post-incident review, "which run caused this?" becomes guesswork.

**Do instead:**
```bash
echo "$(date '+%Y-%m-%d %H:%M:%S'): Starting job"
  # Output: 2026-04-16 20:07:00 (sorts correctly, easily parsed)
```

---

### Configure Log Rotation

**Detection**:
```bash
grep -rl "\.log" --include="*.sh" scripts/ | xargs grep -L "mtime\|logrotate\|rotate"
rg -l '\.log' --type sh | xargs rg -L 'mtime|logrotate|rotate'
```

**Signal**:
```bash
LOG="/var/log/myjob.log"
exec >> "$LOG" 2>&1
echo "$(date): Starting"
  # ... no rotation, this file grows forever ...
```

**Why this matters**: A daily cron job writing 1 MB of output fills 365 MB/year into a single file. On a VPS with a small `/var` partition, this causes disk-full failures in other services (nginx, postgres, syslog). The problem is usually noticed at 3am when cron itself fails to write.

**Do instead:**
```bash
  # Option 1: Daily log files (self-rotating by filename)
LOG="/var/log/myjob_$(date +%Y%m%d).log"
  # Add cleanup: find /var/log -name "myjob_*.log" -mtime +30 -delete

  # Option 2: Explicit rotation at end of script
find "$LOG_DIR" -name "*.log" -mtime +30 -delete
```

---

### Redirect stdout to Persistent Log Files

**Detection**:
```bash
grep -rL ">>" --include="*.sh" scripts/ cron/ | xargs grep -l "echo\|printf" 2>/dev/null
rg -L '>>' --type sh | xargs rg -l 'echo|printf' 2>/dev/null
```

**Signal**:
```bash
#!/bin/bash
set -e
echo "Starting at $(date)"
do_work
echo "Done"
  # All output captured by cron daemon, usually discarded or emailed to root
```

**Why this matters**: Cron captures stdout/stderr and emails them. Most systems have cron email disabled or pointing to an unread mailbox. Even when email works, you lose historical logs: only the most recent run's output is potentially available.

**Do instead:**
```bash
exec >> "/var/log/myjob_$(date +%Y%m%d).log" 2>&1
```

---

## Error-Fix Mappings

| Symptom | Root Cause | Fix |
|---------|------------|-----|
| No log files despite cron running | Output goes to cron mail, not files | Add `exec >> "$LOG" 2>&1` |
| Log file disk usage growing unbounded | No rotation configured | Add `find $LOG_DIR -mtime +30 -delete` |
| Timestamp format is `Thu Apr 16 20:07` | Using bare `$(date)` | Use `$(date '+%Y-%m-%d %H:%M:%S')` |
| Log shows stdout but not error messages | `>> log` without `2>&1` | Change to `>> "$LOG" 2>&1` |
| `/var` disk full, cron stops working | Unbounded single log file | Switch to daily log files with rotation |
| Cannot tell which cron instance wrote what | Multiple jobs sharing one log, no prefix | Add script name to log prefix: `[$(basename "$0")]` |

---

## Detection Commands Reference

```bash
# No log file redirection
grep -rL '>> .*log\|tee.*log' --include="*.sh" scripts/ cron/ jobs/

# No timestamps in log output
grep -rl "echo" --include="*.sh" scripts/ | xargs grep -L "date"

# No log rotation
grep -rl "\.log" --include="*.sh" scripts/ | xargs grep -L "mtime\|logrotate\|rotate"

# Bare $(date) without format (locale-dependent output)
grep -rn '\$(date)[^+]' --include="*.sh" scripts/ cron/

# Stdout only (no file redirection)
grep -rL ">>" --include="*.sh" scripts/ cron/ | xargs grep -l "echo\|printf" 2>/dev/null
```
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall