Skip to content

K-Dense-AI/scientific-agent-skills / skills/modal

Modal is a serverless cloud platform for running Python on demand, including on-demand GPUs. Use when deploying or serving AI/ML models, running GPU-accelerated workloads (training, fine-tuning, inference), serving web endpoints, scheduling batch jobs, or scaling Python code to cloud containers with the Modal SDK.

FileKindSize
SKILL.mdskill md17.6 KB
references/api_reference.mdmarkdown8.5 KB
references/examples.mdmarkdown10.3 KB
references/functions.mdmarkdown8.3 KB
references/getting-started.mdmarkdown4.7 KB
references/gpu.mdmarkdown6.5 KB
references/images.mdmarkdown7.3 KB
references/resources.mdmarkdown3.2 KB
references/scaling.mdmarkdown6.5 KB
references/scheduled-jobs.mdmarkdown4.3 KB
references/secrets.mdmarkdown3.9 KB
references/volumes.mdmarkdown6.8 KB
references/web-endpoints.mdmarkdown8.2 KB

references/volumes.md

View on GitHub
# Modal Volumes

Reviewed against SDK 1.6.0 and the official [Volumes guide](https://modal.com/docs/guide/volumes).
Examples with training/model helpers are illustrative; no cloud persistence test was run.

## Table of Contents

- [Overview](#overview)
- [Creating Volumes](#creating-volumes)
- [Mounting Volumes](#mounting-volumes)
- [Reading and Writing Files](#reading-and-writing-files)
- [CLI Access](#cli-access)
- [Commits and Reloads](#commits-and-reloads)
- [Concurrent Access](#concurrent-access)
- [Volumes v2](#volumes-v2)
- [Common Patterns](#common-patterns)

## Overview

Volumes are Modal's distributed file system, optimized for write-once, read-many workloads like storing model weights and distributing them across containers.

Key characteristics:
- Persistent across function invocations and deployments
- Mountable by multiple functions simultaneously
- Background auto-commits every few seconds
- Final commit on container shutdown

## Creating Volumes

### In Code (Lazy Creation)

```python
vol = modal.Volume.from_name("my-volume", create_if_missing=True)
```

### Via CLI

```bash
modal volume create my-volume

# v2 volume (beta)
modal volume create my-volume --version=2
```

### Programmatic v2

```python
vol = modal.Volume.from_name("my-volume", create_if_missing=True, version=2)
```

## Mounting Volumes

Mount volumes to functions via the `volumes` parameter:

```python
vol = modal.Volume.from_name("model-store", create_if_missing=True)

@app.function(volumes={"/models": vol})
def use_model():
    import json
    # Access files at /models/
    with open("/models/config.json") as f:
        config = json.load(f)
```

Mount multiple volumes:

```python
weights_vol = modal.Volume.from_name("weights")
data_vol = modal.Volume.from_name("datasets")

@app.function(volumes={"/weights": weights_vol, "/data": data_vol})
def train():
    ...
```

## Reading and Writing Files

### Writing

```python
@app.function(volumes={"/data": vol})
def save_results(results):
    import json
    import os

    os.makedirs("/data/outputs", exist_ok=True)
    with open("/data/outputs/results.json", "w") as f:
        json.dump(results, f)
    vol.commit()
```

### Reading

```python
@app.function(volumes={"/data": vol})
def load_results():
    import json
    vol.reload()
    with open("/data/outputs/results.json") as f:
        return json.load(f)
```

### Large Files (Model Weights)

```python
@app.function(volumes={"/models": vol}, gpu="L40S")
def save_model():
    import torch
    model = train_model()
    torch.save(model.state_dict(), "/models/checkpoint.pt")
    vol.commit()

@app.function(volumes={"/models": vol}, gpu="L40S")
def load_model():
    import torch
    vol.reload()
    model = MyModel()
    model.load_state_dict(torch.load("/models/checkpoint.pt", weights_only=True))
    return model
```

## CLI Access

```bash
# List files
modal volume ls my-volume
modal volume ls my-volume /subdir/

# Upload files
modal volume put my-volume local_file.txt
modal volume put my-volume local_file.txt /remote/path/file.txt

# Download files
modal volume get my-volume /remote/file.txt local_file.txt

# Delete a volume
modal volume delete my-volume
```

## Commits and Reloads

Modal auto-commits volume changes in the background every few seconds and on container shutdown.

### Explicit Commit

Persist the writer's changes before signaling downstream work. A commit does not
refresh an already mounted reader; that reader must reload after closing its open
volume file handles:

```python
@app.function(volumes={"/data": vol})
def writer():
    with open("/data/file.txt", "w") as f:
        f.write("hello")
    vol.commit()  # Persist; already mounted readers still need reload()
```

### Reload

See changes from other containers:

```python
@app.function(volumes={"/data": vol})
def reader():
    vol.reload()  # Refresh to see latest writes
    with open("/data/file.txt") as f:
        return f.read()
```

Close all file handles first, and do not read/write the mount while `reload()` runs:
it temporarily appears empty to the reloading container. Reload from the mounted
Function; a local SDK call does not refresh a remote container's mount.

## Concurrent Access

### v1 Volumes

- Recommended max 5 concurrent commits
- Last write wins for concurrent modifications of the same file
- Avoid concurrent modification of identical files
- Max 500,000 files (inodes)

### v2 Volumes

- Hundreds of concurrent writers (distinct files)
- No file count limit
- Improved random access performance
- Files strictly smaller than 1 TiB; at most 262,144 files per directory

## Volumes v2

v2 Volumes (beta) offer significant improvements:
The official guide does not yet guarantee against data loss. Retain another copy
of irreplaceable research data; do not treat v2 as the sole durable archive.

| Feature | v1 | v2 |
|---------|----|----|
| Max files | 500,000 | Unlimited |
| Concurrent writes | ~5 | Hundreds |
| Max file size | No documented limit | Less than 1 TiB |
| Random access | Limited | Full support |
| Hard links | No | Yes |

Enable v2:

```python
vol = modal.Volume.from_name("my-vol-v2", create_if_missing=True, version=2)
```

## Common Patterns

### Model Weight Storage

```python
vol = modal.Volume.from_name("model-weights", create_if_missing=True)

# Download once during image build
def download_weights():
    from huggingface_hub import snapshot_download
    snapshot_download("openai-community/gpt2", local_dir="/models/gpt2")
    vol.commit()

image = (
    modal.Image.debian_slim()
    .uv_pip_install("huggingface_hub")
    .run_function(download_weights, volumes={"/models": vol})
)
```

Mount the same Volume in consuming Functions. A cached `run_function` build step
does not rerun if Volume files are later removed; use a dedicated download Function
when population must be checked independently of the Image cache.

### Training Checkpoints

```python
@app.function(volumes={"/checkpoints": vol}, gpu="H100", timeout=86400)
def train():
    for epoch in range(100):
        train_one_epoch()
        torch.save(model.state_dict(), f"/checkpoints/epoch_{epoch}.pt")
        vol.commit()  # Save checkpoint immediately
```

### Shared Data Between Functions

```python
data_vol = modal.Volume.from_name("shared-data", create_if_missing=True)

@app.function(volumes={"/data": data_vol})
def preprocess():
    # Write processed data
    df.to_parquet("/data/processed.parquet")
    data_vol.commit()

@app.function(volumes={"/data": data_vol})
def analyze():
    data_vol.reload()  # Ensure we see latest data
    df = pd.read_parquet("/data/processed.parquet")
    return df.describe()
```

### Performance Tips

- Volumes are optimized for large files, not many small files
- Keep under 50,000 files and directories for best v1 performance
- Use Parquet or other columnar formats instead of many small CSVs
- For truly temporary data, use `ephemeral_disk` instead of Volumes
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 323kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall