K-Dense-AI/scientific-agent-skills / skills/modal
modal
VerifiedModal is a serverless cloud platform for running Python on demand, including on-demand GPUs. Use when deploying or serving AI/ML models, running GPU-accelerated workloads (training, fine-tuning, inference), serving web endpoints, scheduling batch jobs, or scaling Python code to cloud containers with the Modal SDK.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 17.6 KB |
| references/api_reference.md | markdown | 8.5 KB |
| references/examples.md | markdown | 10.3 KB |
| references/functions.md | markdown | 8.3 KB |
| references/getting-started.md | markdown | 4.7 KB |
| references/gpu.md | markdown | 6.5 KB |
| references/images.md | markdown | 7.3 KB |
| references/resources.md | markdown | 3.2 KB |
| references/scaling.md | markdown | 6.5 KB |
| references/scheduled-jobs.md | markdown | 4.3 KB |
| references/secrets.md | markdown | 3.9 KB |
| references/volumes.md | markdown | 6.8 KB |
| references/web-endpoints.md | markdown | 8.2 KB |
references/scaling.md
View on GitHub# Modal Scaling and Concurrency
Reviewed against SDK 1.6.0 and current [scaling](https://modal.com/docs/guide/scale),
[invocation](https://modal.com/docs/guide/function-invocation-methods),
[concurrency](https://modal.com/docs/guide/concurrent-inputs),
[batching](https://modal.com/docs/guide/dynamic-batching) and
[dynamic configuration](https://modal.com/docs/guide/dynamic-function-config) guides.
Workload placeholders below are illustrative.
## Table of Contents
- [Autoscaling](#autoscaling)
- [Configuration](#configuration)
- [Parallel Execution](#parallel-execution)
- [Concurrent Inputs](#concurrent-inputs)
- [Dynamic Batching](#dynamic-batching)
- [Dynamic Autoscaler Updates](#dynamic-autoscaler-updates)
- [Limits](#limits)
## Autoscaling
Modal automatically manages a pool of containers for each function:
- Spins up containers when there's no capacity for new inputs
- Spins down idle containers to save costs
- Scales container compute to zero by default; persistent storage remains separately billed
No configuration needed for basic autoscaling — it works out of the box.
## Configuration
Fine-tune autoscaling behavior:
```python
@app.function(
max_containers=100, # Upper limit on container count
min_containers=2, # Keep 2 warm (reduces cold starts)
buffer_containers=5, # Reserve 5 extra for burst traffic
scaledown_window=300, # Wait 5 min idle before shutting down
)
def handle_request(data):
...
```
| Parameter | Default | Description |
|-----------|---------|-------------|
| `max_containers` | Platform/workspace limit | Hard cap for this Function pool |
| `min_containers` | 0 | Minimum warm containers (costs money even when idle) |
| `buffer_containers` | Server-selected when omitted | Extra idle containers while active |
| `scaledown_window` | Server-selected when omitted | Maximum idle seconds while scaling down |
SDK defaults are `None` for omitted settings, allowing platform defaults. Set values
explicitly when required. Containers can shut down sooner than the scaledown window
when overprovisioned. Warm pools reduce cold starts but cannot guarantee their absence.
### Trade-offs
- Higher `min_containers` = lower latency, higher cost
- Higher `buffer_containers` = less queuing, higher cost
- Lower `scaledown_window` = faster cost savings, more cold starts
## Parallel Execution
### `.map()` — Process Many Inputs
```python
@app.function()
def process(item):
return heavy_computation(item)
@app.local_entrypoint()
def main():
items = list(range(10_000))
results = list(process.map(items))
```
Modal automatically scales containers to handle the workload. Results maintain input order.
### `.map()` Options
```python
# Unordered results (faster)
for result in process.map(items, order_outputs=False):
handle(result)
# Collect errors instead of raising
results = list(process.map(items, return_exceptions=True))
for r in results:
if isinstance(r, Exception):
print(f"Error: {r}")
```
### `.starmap()` — Multi-Argument
```python
@app.function()
def add(x, y):
return x + y
results = list(add.starmap([(1, 2), (3, 4), (5, 6)]))
# [3, 7, 11]
```
### `.spawn()` — Fire-and-Forget
```python
# Returns immediately
call = process.spawn(large_data)
# Check status or get result later
result = call.get()
```
Up to 1 million queued `.spawn()` inputs. Use a deployed Function for durable jobs,
persist the returned call ID, and retrieve outputs within 7 days of completion.
`spawn_map()` submits many inputs but returns **no result handle** in SDK 1.6.0;
each worker must persist its output if it matters.
## Concurrent Inputs
By default, each container handles one input at a time. Use `@modal.concurrent` to handle multiple:
```python
@app.function(gpu="L40S")
@modal.concurrent(max_inputs=10)
async def predict(text: str):
result = await model.predict_async(text)
return result
```
This is ideal for I/O-bound workloads or async inference where a single GPU can handle multiple requests.
Synchronous functions use threads; async functions share an event loop. Avoid blocking
that loop. `target_inputs` can be set below `max_inputs` to give autoscaling headroom.
### With Web Endpoints
```python
@app.function(gpu="L40S")
@modal.concurrent(max_inputs=20)
@modal.asgi_app()
def web_service():
return fastapi_app
```
## Dynamic Batching
Collect inputs into batches for efficient GPU utilization:
```python
@app.function(gpu="L40S")
@modal.batched(max_batch_size=32, wait_ms=100)
async def batch_predict(texts: list[str]):
# Called with up to 32 texts at once
embeddings = model.encode(texts)
return list(embeddings)
```
- `max_batch_size` — Maximum inputs per batch
- `wait_ms` — How long to wait for more inputs before processing
- The function receives a list and must return a list of the same length
- Callers submit individual elements, e.g. `batch_predict.remote("text")`; the decorator batches them
- A class with a batched method cannot contain other Modal methods; validate the selected configuration before enabling concurrency on a batched workload
## Dynamic Autoscaler Updates
Adjust autoscaling at runtime without redeploying:
```python
@app.function()
def scale_up_for_peak():
process = modal.Function.from_name("my-app", "process")
process.update_autoscaler(min_containers=10, buffer_containers=20)
@app.function()
def scale_down_after_peak():
process = modal.Function.from_name("my-app", "process")
process.update_autoscaler(min_containers=1, buffer_containers=2)
```
Settings revert to the decorator values on the next deployment.
`with_options()`, `with_concurrency()` and `with_batching()` instead create variants
with separate container pools. A base `max_containers` does not cap the sum of variants;
avoid high-cardinality configurations. Variants ignore base `min_containers`, and
`with_options()` does not accept that parameter.
## Limits
| Resource | Limit |
|----------|-------|
| Queued synchronous inputs | 2,000 |
| Total synchronous inputs (running + queued) | 25,000 |
| Pending `.spawn()` inputs | 1,000,000 |
| Concurrent inputs per `.map()` | 1,000 |
| Containers per Function | 4,000, subject to lower workspace/GPU quotas |
| Baseline synchronous invocation rate | 200/s |
| Baseline asynchronous invocation rate | 1,500/s |
These are documented platform limits at review time, not guaranteed capacity.
SDK calls can raise `modal.exception.ResourceExhaustedError`; Web Functions can
return HTTP 429. Use bounded backoff and idempotent work, and check workspace quotas.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |