heymegabyte/agent-skills / 08-deploy-and-runtime-verification
deploy-and-runtime-verification
UnverifiedMANDATORY deploy after every code change. Typecheck → deploy → purge CDN → E2E on production → visual verify → fix-forward loop. Workers Builds native CI/CD, D1 Time Travel PIT recovery, D1→R2 long-term backups, wrangler rollback, wrangler secrets management, structured observability, cross-browser smoke tests, rollback procedures, and GitHub auto-configuration.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 5.6 KB |
| backup-and-disaster-recovery.md | markdown | 2.7 KB |
| changelog-and-releases.md | markdown | 5.3 KB |
| ci-cd-pipeline.md | markdown | 7.1 KB |
| critical-css.md | markdown | 4.7 KB |
| font-subsetting.md | markdown | 4.5 KB |
| gh-fix-ci.md | markdown | 3.5 KB |
| launch-day-sequence.md | markdown | 5.2 KB |
| pipeline-health-check.md | markdown | 4.5 KB |
| r2-lifecycle.md | markdown | 5.6 KB |
| service-worker.md | markdown | 9.7 KB |
| uptime-and-health.md | markdown | 6.5 KB |
pipeline-health-check.md
View on GitHub# Pipeline Health Check (***FIRST ACTION EVERY SESSION INVOLVING BUILDS/JOBS***)
Long-running pipelines (CF Workflows, build queues, generation jobs, container orchestrators) must be inspected for wedged / error rows BEFORE any new trigger. Universal protocol — see `~/.claude/rules/failed-pipeline-protocol.md` for the full rule. This file is the project-sites runbook: exact commands, exact tables, exact thresholds.
## Detect — One-Line D1 Query
```bash
cd <repo> && set -a && source .env.local && set +a
npx wrangler d1 execute project-sites-db-production --remote --env production --json --command \
"SELECT id, slug, status, datetime(updated_at) AS u FROM sites WHERE status IN ('building','queued','generating','imaging','uploading','collecting') AND updated_at < datetime('now','-30 minutes') AND deleted_at IS NULL ORDER BY updated_at"
```
- Any row returned = wedged
- If zero rows: pipeline is healthy, proceed
- **Cron unsticker check:** dashboard → Workers → project-sites → Triggers → cron `*/30 * * * *` last-run timestamp must be ≤30min old
- **Workflow instance status** (when needed): `instance.status()` from worker code OR `wrangler workflows instances describe <workflow> <id> --env production`
## Diagnose — Five Canonical Failure Modes
Order by historical frequency at project-sites:
1. **Schema CHECK constraint silently rejecting status writes** — `SELECT sql FROM sqlite_master WHERE name='sites'` → confirm allowed values include EVERY status the code writes (`draft|queued|collecting|imaging|generating|building|uploading|published|error|archived`). If missing any, table needs recreation.
2. **No hard wall-time cap in workflow** — `apps/project-sites/src/workflows/site-generation.ts` → `MAX_POLLS × interval` must be ≤30min and must call `updateSiteStatus(..., 'error')` + `notifyBuildFailure()` on cap hit.
3. **Cron unsticker threshold drift** — `apps/project-sites/src/index.ts` cron handler must use `datetime('now','-30 minutes')`, never -60.
4. **Missing failure notification** — every error path in workflow must `await notifyBuildFailed({ email, siteName, slug, reason })` from `services/notifications.ts`.
5. **Container / orchestrator OOM / credit-balance** — Anthropic credit hold, R2 quota, container OOM. `audit_logs WHERE site_id=<id> ORDER BY created_at DESC LIMIT 50` shows the last action before silence.
## Fix — Schema Recreation Pattern (D1 has no `ALTER CHECK`)
```sql
PRAGMA foreign_keys=OFF;
CREATE TABLE sites_new ( /* full schema with expanded CHECK */ );
INSERT INTO sites_new SELECT <every column in order> FROM sites;
DROP TABLE sites;
ALTER TABLE sites_new RENAME TO sites;
PRAGMA foreign_keys=ON;
```
Run via `wrangler d1 execute project-sites-db-production --remote --env production --file /tmp/migrate.sql`. Always test on local D1 first (`--local`).
## Verify Fix — Status Write Smoke Test
```bash
npx wrangler d1 execute project-sites-db-production --remote --env production --command \
"UPDATE sites SET status='error' WHERE id='<test-id>'; SELECT status FROM sites WHERE id='<test-id>'"
```
Must return `error`, not silently fail. Then deploy: `cd apps/project-sites && npx wrangler deploy --env production` (sandbox often blocks log writes — use `dangerouslyDisableSandbox: true`).
## Retrigger — Session Mint + Direct Worker URL
- Public hostname `https://projectsites.dev` triggers CF Bot Fight challenge for scripted POSTs → use direct worker URL `https://project-sites.manhattan.workers.dev`
- Reference one-shot script: `/tmp/claude/retrigger-builds.mjs` — mints session via D1 INSERT (sha256 token_hash), POSTs `/api/sites/<id>/reset` with Bearer token
- Always pair with a background monitor task polling D1 every 60s, max 35min
## Operator Stance
This work IS the business. Detect → diagnose → fix → verify → retrigger → monitor → report URLs + codes in one session, no handoff. Document new failure modes in this file the same prompt they occur. Stale runbook = bug.
## Canonical Incident: 2026-05-01 (4 wedged sites, 15+ hours)
**Root cause:** `sites.status` CHECK allowed only 5 values, code wrote 10. Every error-path write silently rejected → cron saw fresh heartbeat `updated_at` → never fired unsticker. Compounded by 60min threshold + no `MAX_POLLS` cap + no `notifyBuildFailed` at error paths.
**Fix bundle:**
- Expanded CHECK
- `MAX_POLLS=30`
- Cron 30min
- Email at every error path
- Direct worker URL for retrigger
**Lesson:** when wedge persists past SLA, FIRST suspect schema silently rejecting writes. `sqlite_master` query is cheap, the answer is usually there.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |