Skip to content

ericrisco/rsc-harness / skills/runpod

runpod

Verified

Use when running GPU compute on RunPod and deciding between Pods (hourly, always-on) and Serverless (per-second, autoscaling) for training, fine-tuning or inference — serverless endpoints, handler workers, worker Docker templates, network volumes, timeout and worker-count tuning, cold starts, and runaway bills. NOT Python-native serverless GPU with snapshot autoscaling (that is `modal`), NOT calling hosted prebuilt model APIs (that is `replicate`), NOT pulling weights from the Hub (that is `huggingface`).

RunPod: GPU compute, two products, one bill

RunPod sells GPU time two ways and they bill on opposite philosophies. Get the choice wrong and you either pay a steep premium for idle work or you pay 24/7 for a box that sits warm doing nothing. Everything below is the RunPod-specific operational playbook: which product a workload belongs on, how to write a worker that does not waste cold-start seconds, and which knobs actually move the number on the invoice.

The two products:

  • Pods — rent a GPU container by the hour. It runs continuously while it is up, billed every hour whether busy or idle. Your dev box, your training job, your Jupyter.
  • Serverless — per-second autoscaling workers. Billed only while a worker is actually running a job, from worker start to full stop, rounded up to the second. Costs roughly 2-3x the equivalent hourly pod rate, but idle gaps cost nothing on flex workers.

RunPod charges zero egress/ingress fees — bandwidth in and out is free, unlike the hyperscalers. That removes one variable from cost math: you only reason about GPU-seconds and storage.

When to use

  • Deploying inference as a Serverless endpoint (vLLM LLM serving, image gen, custom model).
  • Running a training / fine-tuning job or a Jupyter / dev box on a GPU Pod.
  • Writing or debugging a serverless worker handler (handler(job), async, streaming).
  • Building a custom worker Docker template / image, or wiring a network volume.
  • Cost control on RunPod: cold starts, idle bleed, runaway max workers, idle volume charges.
  • Picking a GPU SKU and the pod-vs-serverless tradeoff for a given workload.

When NOT to use

  • Python-native serverless GPU with snapshot autoscaling and no Dockerfile → that is modal.
  • Calling hosted, prebuilt model endpoints you do not host → that is replicate.
  • Pulling/pushing weights, datasets, model cards on the Hub → that is huggingface.
  • Running models locally on your own machine → that is ollama.
  • Provider-agnostic cross-cloud spend dashboards → that is cost-tracking.
  • Generic container packaging → that is docker (here we cover only the RunPod image shape).

Decision: Pod vs Serverless

Workload shapePickWhy
Training / fine-tuning, multi-hour runsPodServerless premium + the 600s default execution timeout kill long jobs. You want the box continuously.
Interactive dev / Jupyter / notebooksPodYou need it now and responsive; per-second autoscaling adds cold-start latency for nothing.
Bursty inference with real idle gapsServerless (flex)Idle costs nothing on flex; you pay only for the seconds a request runs.
24/7 steady high-QPS inferenceCompareActive serverless (40% off flex) vs a dedicated Pod. Past roughly 60% utilization a Pod usually wins.

Rule: if the GPU would sit busy more than ~60% of the time, a Pod is cheaper than serverless even with active-worker discount — model the two before committing.

GPU SKU quick-pick (2026 pod $/hr, lower on Community Cloud)

GPUVRAMPod ~$/hrUse for
L424GB$0.39small models, light inference
A4048GB$0.44mid-size inference, budget training
RTX 409024GB$0.697B-class inference, fast/cheap
L40S48GB$0.8613B inference, image gen
A100 80GB80GB$1.39training, large-batch inference
H100 PCIe80GB$2.89the biggest models / fastest training

Rule: pick the smallest GPU the model fits in VRAM. Defaulting to H100 is up to ~7x the cost for zero speedup when the workload is memory-bound and fits on an L40S or 4090.

Serverless worker handler

A worker is a Python file using the runpod SDK. The minimum: a function that reads job["input"], returns a dict, and is registered with runpod.serverless.start.

import runpod

def handler(job):
    job_input = job["input"]
    prompt = job_input["prompt"]
    # ... run the model ...
    return {"output": f"echo: {prompt}"}

runpod.serverless.start({"handler": handler})

Async handler (for awaiting model calls) and a streaming generator both work:

import runpod

async def handler(job):
    job_input = job["input"]
    return {"output": await run_model(job_input)}

# Streaming: yield chunks from an async generator instead of returning once.
async def stream_handler(job):
    async for token in generate(job["input"]["prompt"]):
        yield {"token": token}

runpod.serverless.start({"handler": stream_handler, "return_aggregate_stream": True})

Test locally before you push an image. A broken handler still burns build minutes and cold-start seconds when discovered on the platform.

# One-shot: reads ./test_input.json, runs the handler once, prints output.
python worker.py

# HTTP server emulating the real endpoint at http://localhost:8000.
python worker.py --rp_serve_api

Concurrency (concurrency_modifier), job cancel, refresh-worker, and the run / runsync / stream / status / cancel / health HTTP endpoints live in references/serverless-workers.md.

Templates & images

A custom template is a Docker image plus environment variables. Pin the base; an unpinned or :latest base re-pulls on cold start and lengthens it.

# Pin the CUDA base — never bare :latest.
FROM runpod/base:0.6.2-cuda12.4.1

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY handler.py .

# The worker entrypoint runs the handler.
CMD ["python", "-u", "handler.py"]

vLLM shortcut. For OpenAI-compatible LLM serving, use the prebuilt worker-vllm image instead of writing a handler. Its AsyncEngineArgs are set via UPPERCASE env vars:

# Template env vars on the endpoint — these map to vLLM AsyncEngineArgs.
MODEL_NAME=mistralai/Mistral-7B-Instruct-v0.3
MAX_MODEL_LEN=8192

Network volumes

Attach a network volume when data outgrows what you want in the image:

  • Datasets larger than container disk.
  • Weights shared across many workers (download once, mount everywhere).
  • Checkpoints that must survive a worker / pod restart.

Two traps to state up front:

  1. A volume locks the endpoint/pod to one data center and adds network latency. That shrinks the pool of available GPUs in that DC — you can get stuck waiting for capacity.
  2. Idle volume cost is double. Pod volume disk bills ~$0.10/GB/mo while running but ~$0.20/GB/mo while the pod is stopped. A forgotten volume on a stopped pod bleeds money.

Rule: bake small, static weights into the image; reserve volumes for large mutable data (datasets, checkpoints). Full storage tables are in references/cost-and-scaling.md.

Cost control — the four knobs that move the bill

KnobDefaultWhat it does
Idle Timeout5sHow long a worker stays warm (and billed) after a job. Lower for spiky traffic; raise to dodge repeated cold starts.
Execution Timeout600sMax single-job duration (range 5s–7 days). Set it so a hung job cannot run for days.
Max Workers—Your concurrency cap and cost ceiling. Never leave it sky-high; set ~20% over expected peak.
Active Workers0Always-warm minimum: zero cold start but billed 24/7 (at ~40% off the flex rate). Use only when a latency SLA demands it.

Plus FlashBoot: enable it on flex workers to cut cold start (model load into GPU memory) toward sub-200ms by caching, so flex stops feeling slow.

Worked example — 100k requests/day, 2s each on RTX 4090 serverless ($1.10/hr equiv):

  • Compute is ~55.5 GPU-hours/day regardless of mode → roughly $61/day of actual work.
  • Flex adds idle-timeout tails per cold worker but nothing during true idle — best for bursty business-hours traffic.
  • Active workers remove cold starts but bill 24/7; only worth it if traffic is steady enough that the 40% discount beats paying for idle.
  • A dedicated Pod at ~$0.69/hr = ~$497/mo flat — wins only if utilization stays high.

Full active-vs-flex math and monthly scenarios: references/cost-and-scaling.md.

Driving resources

runpodctl is the open-source CLI. It outputs JSON by default (agent-friendly); add --output table or --output yaml for humans. Pods ship with it pre-installed using a pod-scoped key.

runpodctl serverless list                 # JSON by default
runpodctl serverless get <endpoint-id>
runpodctl serverless update <endpoint-id> --output table
runpodctl get pod --output table

Python SDK for programmatic control — key from env, never in source:

import os, runpod

runpod.api_key = os.environ["RUNPOD_API_KEY"]  # never a literal

pod = runpod.create_pod(name="train", image_name="my/img:1.0", gpu_type_id="NVIDIA A100 80GB PCIe")
runpod.stop_pod(pod["id"])      # also resume_pod / terminate_pod

ep = runpod.Endpoint("<endpoint-id>")
job = ep.run({"prompt": "hi"})  # async: job.status(), job.output()
out = ep.run_sync({"prompt": "hi"})  # blocks, ~90s max

Rule: read the API key from RUNPOD_API_KEY. A leaked key is a stranger spending on your GPUs.

Anti-patterns

BadGoodWhy
Max Workers left unbounded / sky-highBound it ~20% over peakOne traffic spike scales to an unbounded bill.
H100 by defaultSmallest GPU that fits VRAM~7x cost for zero speedup when the model fits an L40S/4090.
6-hour training run on ServerlessRun it on a PodServerless premium + 600s execution timeout kills long jobs.
Hardcoded API key in sourceos.environ["RUNPOD_API_KEY"]A leaked key = a stranger's GPU bill on your card.
Weights downloaded at cold startBake into image or mount a volumeEvery cold start re-downloads and pays for the wait.
Active workers "just in case"Flex + FlashBootActive bills 24/7; FlashBoot makes flex cold starts cheap.
Volume left on a stopped podDetach / delete when idleIdle volume bills ~$0.20/GB/mo silently.
Push image, debug on the platformpython worker.py --rp_serve_api firstDebugging on cold-start seconds is slow and costs money.

Verify

Run bash scripts/verify.sh <worker-dir> to statically lint a worker directory: it checks for runpod.serverless.start, a handler with a return/yield, no hardcoded API key, a bounded max_workers plus timeout keys in any config, and a pinned FROM + CMD in a Dockerfile. Pure grep/parse — no network, no RunPod account. It exits 0 on an empty dir.

writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 319kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall