Skip to content

heymegabyte/agent-skills / 08-deploy-and-runtime-verification

deploy-and-runtime-verification

Unverified

MANDATORY deploy after every code change. Typecheck → deploy → purge CDN → E2E on production → visual verify → fix-forward loop. Workers Builds native CI/CD, D1 Time Travel PIT recovery, D1→R2 long-term backups, wrangler rollback, wrangler secrets management, structured observability, cross-browser smoke tests, rollback procedures, and GitHub auto-configuration.

FileKindSize
SKILL.mdskill md5.6 KB
backup-and-disaster-recovery.mdmarkdown2.7 KB
changelog-and-releases.mdmarkdown5.3 KB
ci-cd-pipeline.mdmarkdown7.1 KB
critical-css.mdmarkdown4.7 KB
font-subsetting.mdmarkdown4.5 KB
gh-fix-ci.mdmarkdown3.5 KB
launch-day-sequence.mdmarkdown5.2 KB
pipeline-health-check.mdmarkdown4.5 KB
r2-lifecycle.mdmarkdown5.6 KB
service-worker.mdmarkdown9.7 KB
uptime-and-health.mdmarkdown6.5 KB

pipeline-health-check.md

View on GitHub
# Pipeline Health Check (***FIRST ACTION EVERY SESSION INVOLVING BUILDS/JOBS***)

Long-running pipelines (CF Workflows, build queues, generation jobs, container orchestrators) must be inspected for wedged / error rows BEFORE any new trigger. Universal protocol — see `~/.claude/rules/failed-pipeline-protocol.md` for the full rule. This file is the project-sites runbook: exact commands, exact tables, exact thresholds.

## Detect — One-Line D1 Query

```bash
cd <repo> && set -a && source .env.local && set +a
npx wrangler d1 execute project-sites-db-production --remote --env production --json --command \
  "SELECT id, slug, status, datetime(updated_at) AS u FROM sites WHERE status IN ('building','queued','generating','imaging','uploading','collecting') AND updated_at < datetime('now','-30 minutes') AND deleted_at IS NULL ORDER BY updated_at"
```

- Any row returned = wedged
- If zero rows: pipeline is healthy, proceed
- **Cron unsticker check:** dashboard → Workers → project-sites → Triggers → cron `*/30 * * * *` last-run timestamp must be ≤30min old
- **Workflow instance status** (when needed): `instance.status()` from worker code OR `wrangler workflows instances describe <workflow> <id> --env production`

## Diagnose — Five Canonical Failure Modes

Order by historical frequency at project-sites:

1. **Schema CHECK constraint silently rejecting status writes** — `SELECT sql FROM sqlite_master WHERE name='sites'` → confirm allowed values include EVERY status the code writes (`draft|queued|collecting|imaging|generating|building|uploading|published|error|archived`). If missing any, table needs recreation.
2. **No hard wall-time cap in workflow** — `apps/project-sites/src/workflows/site-generation.ts` → `MAX_POLLS × interval` must be ≤30min and must call `updateSiteStatus(..., 'error')` + `notifyBuildFailure()` on cap hit.
3. **Cron unsticker threshold drift** — `apps/project-sites/src/index.ts` cron handler must use `datetime('now','-30 minutes')`, never -60.
4. **Missing failure notification** — every error path in workflow must `await notifyBuildFailed({ email, siteName, slug, reason })` from `services/notifications.ts`.
5. **Container / orchestrator OOM / credit-balance** — Anthropic credit hold, R2 quota, container OOM. `audit_logs WHERE site_id=<id> ORDER BY created_at DESC LIMIT 50` shows the last action before silence.

## Fix — Schema Recreation Pattern (D1 has no `ALTER CHECK`)

```sql
PRAGMA foreign_keys=OFF;
CREATE TABLE sites_new ( /* full schema with expanded CHECK */ );
INSERT INTO sites_new SELECT <every column in order> FROM sites;
DROP TABLE sites;
ALTER TABLE sites_new RENAME TO sites;
PRAGMA foreign_keys=ON;
```

Run via `wrangler d1 execute project-sites-db-production --remote --env production --file /tmp/migrate.sql`. Always test on local D1 first (`--local`).

## Verify Fix — Status Write Smoke Test

```bash
npx wrangler d1 execute project-sites-db-production --remote --env production --command \
  "UPDATE sites SET status='error' WHERE id='<test-id>'; SELECT status FROM sites WHERE id='<test-id>'"
```

Must return `error`, not silently fail. Then deploy: `cd apps/project-sites && npx wrangler deploy --env production` (sandbox often blocks log writes — use `dangerouslyDisableSandbox: true`).

## Retrigger — Session Mint + Direct Worker URL

- Public hostname `https://projectsites.dev` triggers CF Bot Fight challenge for scripted POSTs → use direct worker URL `https://project-sites.manhattan.workers.dev`
- Reference one-shot script: `/tmp/claude/retrigger-builds.mjs` — mints session via D1 INSERT (sha256 token_hash), POSTs `/api/sites/<id>/reset` with Bearer token
- Always pair with a background monitor task polling D1 every 60s, max 35min

## Operator Stance

This work IS the business. Detect → diagnose → fix → verify → retrigger → monitor → report URLs + codes in one session, no handoff. Document new failure modes in this file the same prompt they occur. Stale runbook = bug.

## Canonical Incident: 2026-05-01 (4 wedged sites, 15+ hours)

**Root cause:** `sites.status` CHECK allowed only 5 values, code wrote 10. Every error-path write silently rejected → cron saw fresh heartbeat `updated_at` → never fired unsticker. Compounded by 60min threshold + no `MAX_POLLS` cap + no `notifyBuildFailed` at error paths.

**Fix bundle:**

- Expanded CHECK
- `MAX_POLLS=30`
- Cron 30min
- Email at every error path
- Direct worker URL for retrigger

**Lesson:** when wedge persists past SLA, FIRST suspect schema silently rejecting writes. `sqlite_master` query is cheap, the answer is usually there.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall