supercheck-io/supercheck / .agents/skills/execution-engine
supercheck-execution-engine
UnverifiedWork on Supercheck BullMQ queues, schedulers, capacity management and worker admission, Playwright or k6 execution, runs stuck queued or stalled, monitors, dynamic locations, cancellation, Redis, Kubernetes Jobs, gVisor, or app-worker execution contracts.
Supercheck execution engine
flowchart LR
TRIGGER[Manual, schedule, webhook, CLI] --> GATE[Atomic capacity reservation]
GATE --> Q[(Regional or global BullMQ queue)]
Q --> W[NestJS worker]
W --> JOB[Kubernetes Job]
JOB --> GV[gVisor sandbox]
GV --> RESULT[Database and object storage]
RESULT --> RELEASE[Terminal state and capacity release]
Start here
| File | Why |
|---|---|
app/src/lib/capacity-manager.ts | Atomic Redis reserve/release (app-side admission) |
worker/src/execution/services/execution-admission.service.ts, execution-queue-fairness.service.ts | Worker-side admission and fairness, the enforcement point |
app/src/lib/queue.ts, app/src/lib/job-scheduler.ts, app/src/lib/monitor-scheduler.ts | Queues, connections, schedulers |
app/src/lib/status-page-scheduler.ts, app/src/lib/sre/incident-renotify.ts, app/src/sre/lib/background-alert-triage-processor.ts | App-owned BullMQ workers (status-page-jobs, sre-incident-renotify, sre-alert-triage); closeQueue() in queue.ts closes each on SIGTERM, so new ones need a shutdown hook there and an entry in the Bull dashboard and worker queue alerting |
app/src/lib/location-registry.ts, app/src/lib/monitor-location-routing.ts | Location normalization and queue routing |
worker/src/common/security/container-executor.service.ts (+ .spec.ts) | Kubernetes Job rendering: runtime class, deadlines, TTL |
Workload model
- App routes/actions/schedulers admit and enqueue Playwright, k6, and monitor work.
- Scheduler processors execute in the app tier so scheduled work passes through central capacity policy. Workers must not consume scheduler payloads.
- Worker processors under
worker/src/execution,worker/src/k6, andworker/src/monitorrun independently from the app release. - Regional queues target configured locations; global queues are used only when any eligible worker may process the job.
- Queue constants, payload types, status enums, retries, timeout meanings, IDs, and result contracts must remain in app/worker parity.
Capacity and idempotency
- Organization-level admission is the capacity source of truth. Running and queued reservations use atomic Redis operations and fail closed if capacity state cannot be trusted. App-side checks are advisory; worker admission enforces.
- Reserve before dispatch. Release exactly once on every terminal path: completion, terminal failure, cancellation, timeout, dispatch failure, and unrecoverable worker loss.
- Retryable BullMQ attempts retain the reservation until the job becomes terminal.
- Duplicate delivery, scheduler overlap, API retries, and webhook retries must not create duplicate executions or double-release capacity.
- Keep configured app capacity, worker replicas, per-processor concurrency, and Kubernetes resource ceilings coherent.
Redis and BullMQ
- Redis must use
maxmemory-policy noeviction; evicting queue keys can lose or corrupt jobs. - General queue operations may share appropriate base connections.
- Every
QueueEventsconsumer needs its own dedicated connection because blockingXREADcannot share a normal command connection. In the app, pass Redis options (buildRedisOptions(...)) so BullMQ creates and owns it; an externalduplicate()gets duplicated again and leaks. - Do not set
commandTimeouton shared options inherited by QueueEvents (see the note inapp/src/lib/queue.ts); non-blocking clients such as the stalled-job handler may set it. Keep bounded connection establishment while allowing intentional failover reconnection. - Startup scheduling runs under a Redis init lock and must be atomic and idempotent across replicas. Data-lifecycle cleanups use
upsertJobScheduler(id<entity>-cleanup, same template name as the legacy repeatable, so an old replica's remove-then-add leaves one schedule); startup removes the legacy<entity>-cleanup-recurringrepeatable once. The data-lifecycleQueueEventsis created lazily on the first manual cleanup and closed at shutdown. - Scheduled jobs (Playwright and k6) live in the single
job-schedulerqueue; the processor branches on the DBjobType.scheduleJobalways adds there and removes the job's repeatable from both queues;deleteScheduledJobclears both;initializeJobSchedulersmoves leftoverk6-job-schedulerrepeatables intojob-scheduler. Thek6-job-schedulerWorker, dashboard and alerting entries stay for one release so old-replica ticks still fire once (shortcut: legacy k6-job-scheduler consumer kept for one release; remove in release N+1). The legacy Worker's processor (app/src/lib/scheduler/legacy-k6-tick.ts) converges a schedule an old-version replica re-adds there without losing a due fire: ifjob-schedulerholds exactly that one schedule, the legacy repeatable is removed and the tick is skipped only when that copy's next fire is the tick's own slot; otherwise the database cron (looked up within the tick's project and organization) decides,job-scheduleris made to hold one matching repeatable (added before any other is removed, keeping a matching entry's timezone), and the tick runs. A retry re-runs this convergence. Both queues give a slot the same tick id, so the per-tick run guard below rejects a second run when thejob-schedulercopy fired first. - A periodic repair sweep (
repairJobSchedules, every 5 min, its own Redis lock, skipped while the init lock is held so init never backs off, 100 jobs per sweep from a rotating cursor, re-reading each row so a concurrent cron edit wins) schedules live cron jobs outside pending organizations and deleted projects whosescheduled-job-<id>repeatable is missing in both queues, so a failed restore, a cancelled organization deletion or an orphan cleanup that raced a cancel heals without a restart. - With no
commandTimeout, commands wait in ioredis's offline queue during an outage. Enqueue entry points (theadd*ToQueueproducers, the execution rate limiter, the API-key trigger route and the scheduler tick) callassertExecutionQueueReady()/ checkstatus === 'ready'before any run row or reservation, (a client that is connecting or reconnecting, i.e. a cold start or a 1-3 s Sentinel failover, gets one shared bounded 2 s wait,app/src/lib/redis-ready.ts; only an ended client fails at once, andcloseis not treated as terminal because ioredis emits it before every retry) so the request gets the existing 503 (EXECUTION_QUEUE_UNAVAILABLE/EXECUTION_ADMISSION_UNAVAILABLE) instead of hanging. - Promotion deletes nothing it still needs: the payload
capacity:job:<id>gets a short expiry in the promotion script and is deleted afterqueue.add, so a crash in between lets the worker's orphan sweep recover the run (once it is past its 30-minute age bar; the payload's 15-minute grace also covers aqueue.addstalled by a failover) instead of after 48h. The drift reconciler corrects the running counter by compare-and-set against the value it read, so a concurrent INCR/DECR is never overwritten. - A scheduler tick is idempotent: its run carries
metadata.scheduleTickKey(<repeat job id>:<attemptsMade>) and a redelivered tick that already created a run is skipped, and the check-and-insert runs under a transaction-scoped advisory lock on the tick key (BullMQ only redelivers after the lock expired, but a processor that lost its lock keeps running, so two can overlap); a BullMQ retry (higherattemptsMade) is a new tick on purpose. A tick that finds Redis not ready fails before creating a run and is retried by BullMQ (3 attempts, ~15s of backoff); a longer outage loses that cron run. Usage is metered per attempt (runId:attempt:N). - Monitor connection count across app replicas, workers, schedulers, dashboards, and event streams.
Payloads, secrets, and artifacts
- Queue payloads contain trusted identifiers and bounded safe execution inputs—not decrypted tenant secrets.
- Workers resolve current scoped variables immediately before execution and redact values from output.
- Large scripts/results/artifacts belong in bounded payload/storage paths, not oversized Redis values or unbounded process memory.
- Artifact upload failures have explicit timeout/retry behavior and cannot leave runs permanently active.
Cancellation and failures
- Cancellation is idempotent: mark intent, stop queued work or signal active work, terminate the execution Job when necessary, persist one terminal state, and release capacity once.
- Runs cancelled before any worker picks them up must still be finalized, and give-up paths (for example capacity-wait timeouts) must not overwrite an existing cancellation marker. Cancellation is asynchronous to API callers and E2E tests.
- Capacity and unverifiable-execution give-up paths use
DbService.failRunUnlessCancelled; its single guarded write prevents a late cancellation from being overwritten and preserves terminal results. - Let BullMQ retry eligible failures; do not swallow processor errors after marking an intermediate state. Once an execution container has run, a post-run failure (report upload, result save) is
UnrecoverableError: a retry would run the test against the customer's target again. - A busy pod throws
ExecutionCapacityErrorbefore any reservation so the processor delays the job (deferForCapacity) without spending an attempt. - Stalled-job policy is bounded: Playwright and k6 workers use a 5-minute
lockDuration, while monitor workers use 15 minutes to allow for slow synthetic checks. Locks are renewed while workers are alive, so a crashed worker frees its run quickly, and the age-based stalled-run sweep stays above the 60-minute maximum run. - Worker shutdown stops queue intake and reports not-ready, then drains for
WORKER_SHUTDOWN_DRAIN_MS(max 65 minutes) while in-flight work finishes. The drain counts everything picked up (trackInFlight), not only runs that already have a Job, and nothing creates a Job after it ends. Remaining Jobs are then deleted. - A restart-interrupted run is
interrupted, never a user cancellation. If its pod never started it is delayed withExecutionCapacityError(no attempt, no billing) and another worker runs it. If it had started it is terminal: not re-run, stored with "Interrupted by worker restart" and each engine's existing error status (Playwrightfailed, k6error), and it sends no job notification or alert (treated likecancelledbyhandleJobNotifications). - Teardown that in-flight runs depend on (the shared Redis client, SMTP transport, queue workers) runs in
onApplicationShutdown, after the drain.ContainerExecutorService.onModuleDestroyowns the drain: it stops intake, drains in-flight work, clears cancellation timers and deletes the remaining Jobs. OtheronModuleDestroyhooks only stop intake and timers. Queue workers close only after the drain.
Production sandbox
- Production execution runs in a dedicated namespace with
runtimeClassName: gvisorand fails closed if required isolation is unavailable. EXECUTION_RUNTIME_CLASS_NAME=noneis local-development only.- Jobs use the zero-permission execution service account, non-root identity, read-only root filesystem, dropped capabilities, no privilege escalation, bounded writable storage, CPU/memory/ephemeral limits, and deadline/cleanup settings.
- Namespace LimitRange/ResourceQuota and worker-configured ceilings stay aligned.
- NetworkPolicy denies by default and allows only required DNS/egress. Application SSRF protection remains required even with network isolation.
Verification
- Test races at capacity boundaries, duplicate dispatch, retries, cancellation timing, Redis loss/recovery, worker loss, timeout, upload failure, and cleanup.
- Test rolling app/worker compatibility and location normalization/routing.
- Render the actual Kubernetes Job and security policy; source-template review alone is insufficient.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |