davila7/claude-code-templates / cli-tool/components/skills/ai-research/infrastructure-skypilot
infrastructure-skypilot
FlaggedMulti-cloud orchestration for ML workloads with automatic cost optimization. Use when you need to run training or batch jobs across multiple clouds, leverage spot instances with auto-recovery, or optimize GPU costs across providers.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.4 KB |
| references/advanced-usage.md | markdown | 7.3 KB |
| references/troubleshooting.md | markdown | 10.2 KB |
references/advanced-usage.md
View on GitHub# SkyPilot Advanced Usage Guide
## Multi-Cloud Strategies
### Cloud selection patterns
```yaml
# Prefer specific clouds in order
resources:
accelerators: A100:8
any_of:
- cloud: gcp
region: us-central1
- cloud: aws
region: us-west-2
- cloud: azure
region: westus2
```
### Wildcard regions
```yaml
resources:
cloud: aws
region: us-* # Any US region
accelerators: A100:8
```
### Kubernetes + Cloud fallback
```yaml
resources:
accelerators: A100:8
any_of:
- cloud: kubernetes
- cloud: aws
- cloud: gcp
```
## Advanced Resource Configuration
### Instance type constraints
```yaml
resources:
instance_type: p4d.24xlarge # Specific instance
# OR
cpus: 32+
memory: 128+
accelerators: A100:8
```
### Disk configuration
```yaml
resources:
disk_size: 500 # GB
disk_tier: best # low, medium, high, ultra, best
```
### Network tier
```yaml
resources:
network_tier: best # High-performance networking
```
## Production Managed Jobs
### Job configuration
```yaml
name: production-training
resources:
accelerators: H100:8
use_spot: true
spot_recovery: FAILOVER
# Retry configuration
max_restarts_on_errors: 3
```
### Controller scaling
For large-scale deployments (hundreds of jobs):
```bash
# Increase controller memory
sky jobs launch --controller-resources memory=32
```
### Static credentials
Use non-expiring credentials for controllers:
```bash
# AWS: Use IAM role or long-lived access keys
# GCP: Use service account JSON key
# Azure: Use service principal
```
## Advanced File Mounts
### Git repository workdir
```yaml
workdir:
url: https://github.com/user/repo.git
ref: main
# For private repos, set GIT_TOKEN env var
```
### Multiple storage backends
```yaml
file_mounts:
/data/s3:
source: s3://my-bucket/data
mode: MOUNT
/data/gcs:
source: gs://my-bucket/data
mode: MOUNT
/outputs:
name: training-outputs
store: s3
mode: MOUNT_CACHED
```
### Rsync exclude patterns
```yaml
workdir: .
# Use .skyignore or .gitignore for excludes
```
Create `.skyignore`:
```
__pycache__/
*.pyc
.git/
.env
node_modules/
```
## Distributed Training Patterns
### PyTorch DDP
```yaml
num_nodes: 4
resources:
accelerators: A100:8
run: |
torchrun \
--nnodes=$SKYPILOT_NUM_NODES \
--nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \
--node_rank=$SKYPILOT_NODE_RANK \
--master_addr=$(echo "$SKYPILOT_NODE_IPS" | head -n1) \
--master_port=12355 \
train.py
```
### DeepSpeed
```yaml
num_nodes: 4
resources:
accelerators: A100:8
setup: |
pip install deepspeed
run: |
# Create hostfile
echo "$SKYPILOT_NODE_IPS" | awk '{print $1 " slots=8"}' > /tmp/hostfile
deepspeed --hostfile=/tmp/hostfile \
--num_nodes=$SKYPILOT_NUM_NODES \
--num_gpus=$SKYPILOT_NUM_GPUS_PER_NODE \
train.py --deepspeed ds_config.json
```
### Ray Train
```yaml
num_nodes: 4
resources:
accelerators: A100:8
run: |
# Head node starts Ray head
if [ "${SKYPILOT_NODE_RANK}" == "0" ]; then
ray start --head --port=6379
# Wait for workers
sleep 30
python train_ray.py
else
ray start --address=$(echo "$SKYPILOT_NODE_IPS" | head -n1):6379
fi
```
## Sky Serve Advanced
### Multi-replica serving
```yaml
service:
readiness_probe:
path: /health
initial_delay_seconds: 60
period_seconds: 10
replica_policy:
min_replicas: 2
max_replicas: 20
target_qps_per_replica: 5.0
upscale_delay_seconds: 60
downscale_delay_seconds: 300
load_balancing_policy: round_robin # or least_connections
```
### Blue-green deployment
```bash
# Deploy new version
sky serve up -n my-service-v2 service_v2.yaml
# Test new version
curl https://my-service-v2.skypilot.cloud/health
# Switch traffic (update DNS/load balancer)
# Then terminate old version
sky serve down my-service-v1
```
### Service with multiple accelerator options
```yaml
service:
replica_policy:
min_replicas: 1
max_replicas: 5
resources:
accelerators:
L40S: 1
A100: 1
A10G: 1
any_of:
- cloud: aws
- cloud: gcp
```
## Cost Optimization
### Spot instance strategies
```yaml
resources:
accelerators: A100:8
use_spot: true
spot_recovery: FAILOVER # or FAILOVER_NO_WAIT
# Always checkpoint for spot jobs
file_mounts:
/checkpoints:
name: spot-checkpoints
store: s3
mode: MOUNT_CACHED
```
### Reserved instance hints
```yaml
resources:
accelerators: A100:8
# SkyPilot considers reserved instances in cost calculation
```
### Budget constraints
```bash
# Dry run to see cost estimate
sky launch task.yaml --dryrun
# Set max cluster cost (future feature)
# sky launch task.yaml --max-cost-per-hour 50
```
## Kubernetes Integration
### Using existing clusters
```bash
# Configure kubeconfig
export KUBECONFIG=~/.kube/config
# Verify
sky check kubernetes
```
### Pod configuration
```yaml
resources:
cloud: kubernetes
accelerators: A100:1
config:
kubernetes:
pod_config:
spec:
runtimeClassName: nvidia
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
```
### Multi-cluster
```yaml
resources:
any_of:
- cloud: kubernetes
infra: cluster1
- cloud: kubernetes
infra: cluster2
- cloud: aws
```
## API Server Deployment
### Team setup
```bash
# Start API server
sky api serve --host 0.0.0.0 --port 8000
# Connect clients
sky api login --endpoint https://your-server:8000
```
### Authentication
```bash
# Create service account
sky api create-service-account my-service
# Use token in CI/CD
export SKYPILOT_API_TOKEN=...
sky launch task.yaml
```
## Advanced CLI Patterns
### Parallel cluster operations
```bash
# Launch multiple clusters in parallel
for i in {1..10}; do
sky launch -c cluster-$i task.yaml --detach &
done
wait
```
### Batch job submission
```bash
# Submit many jobs
for config in configs/*.yaml; do
name=$(basename $config .yaml)
sky jobs launch -n $name $config
done
# Monitor all jobs
sky jobs queue
```
### Conditional execution
```yaml
run: |
# Only run on head node
if [ "${SKYPILOT_NODE_RANK}" == "0" ]; then
python main.py
else
python worker.py
fi
```
## Environment Management
### Environment variables
```yaml
envs:
WANDB_PROJECT: my-project
HF_TOKEN: $HF_TOKEN # Inherit from local
CUDA_VISIBLE_DEVICES: "0,1,2,3"
# Secrets (hidden in logs)
secrets:
- WANDB_API_KEY
- HF_TOKEN
```
### Config overrides
```yaml
config:
# Override global config
jobs:
controller:
resources:
memory: 32
```
## Monitoring and Observability
### Log streaming
```bash
# Stream logs
sky logs mycluster
# Follow specific job
sky logs mycluster 1
# Managed job logs
sky jobs logs my-job --follow
```
### Integration with W&B/MLflow
```yaml
envs:
WANDB_API_KEY: $WANDB_API_KEY
WANDB_PROJECT: my-project
run: |
wandb login $WANDB_API_KEY
python train.py --wandb
```
## Debugging
### SSH access
```bash
# SSH to head node
ssh mycluster
# SSH to worker node
ssh mycluster-worker1
# Port forwarding
ssh -L 8080:localhost:8080 mycluster
```
### Interactive debugging
```bash
# Launch interactive cluster
sky launch -c debug --gpus A100:1
# SSH and debug
ssh debug
```
### Job inspection
```bash
# View job queue
sky queue mycluster
# Cancel specific job
sky cancel mycluster 1
# View job details
sky logs mycluster 1
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |