Skip to content

davila7/claude-code-templates / cli-tool/components/skills/ai-research/infrastructure-skypilot

infrastructure-skypilot

Flagged

Multi-cloud orchestration for ML workloads with automatic cost optimization. Use when you need to run training or batch jobs across multiple clouds, leverage spot instances with auto-recovery, or optimize GPU costs across providers.

FileKindSize
SKILL.mdskill md9.4 KB
references/advanced-usage.mdmarkdown7.3 KB
references/troubleshooting.mdmarkdown10.2 KB

references/advanced-usage.md

View on GitHub
# SkyPilot Advanced Usage Guide

## Multi-Cloud Strategies

### Cloud selection patterns

```yaml
# Prefer specific clouds in order
resources:
  accelerators: A100:8
  any_of:
    - cloud: gcp
      region: us-central1
    - cloud: aws
      region: us-west-2
    - cloud: azure
      region: westus2
```

### Wildcard regions

```yaml
resources:
  cloud: aws
  region: us-*  # Any US region
  accelerators: A100:8
```

### Kubernetes + Cloud fallback

```yaml
resources:
  accelerators: A100:8
  any_of:
    - cloud: kubernetes
    - cloud: aws
    - cloud: gcp
```

## Advanced Resource Configuration

### Instance type constraints

```yaml
resources:
  instance_type: p4d.24xlarge  # Specific instance
  # OR
  cpus: 32+
  memory: 128+
  accelerators: A100:8
```

### Disk configuration

```yaml
resources:
  disk_size: 500  # GB
  disk_tier: best  # low, medium, high, ultra, best
```

### Network tier

```yaml
resources:
  network_tier: best  # High-performance networking
```

## Production Managed Jobs

### Job configuration

```yaml
name: production-training

resources:
  accelerators: H100:8
  use_spot: true
  spot_recovery: FAILOVER

# Retry configuration
max_restarts_on_errors: 3
```

### Controller scaling

For large-scale deployments (hundreds of jobs):

```bash
# Increase controller memory
sky jobs launch --controller-resources memory=32
```

### Static credentials

Use non-expiring credentials for controllers:

```bash
# AWS: Use IAM role or long-lived access keys
# GCP: Use service account JSON key
# Azure: Use service principal
```

## Advanced File Mounts

### Git repository workdir

```yaml
workdir:
  url: https://github.com/user/repo.git
  ref: main
  # For private repos, set GIT_TOKEN env var
```

### Multiple storage backends

```yaml
file_mounts:
  /data/s3:
    source: s3://my-bucket/data
    mode: MOUNT

  /data/gcs:
    source: gs://my-bucket/data
    mode: MOUNT

  /outputs:
    name: training-outputs
    store: s3
    mode: MOUNT_CACHED
```

### Rsync exclude patterns

```yaml
workdir: .

# Use .skyignore or .gitignore for excludes
```

Create `.skyignore`:
```
__pycache__/
*.pyc
.git/
.env
node_modules/
```

## Distributed Training Patterns

### PyTorch DDP

```yaml
num_nodes: 4

resources:
  accelerators: A100:8

run: |
  torchrun \
    --nnodes=$SKYPILOT_NUM_NODES \
    --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \
    --node_rank=$SKYPILOT_NODE_RANK \
    --master_addr=$(echo "$SKYPILOT_NODE_IPS" | head -n1) \
    --master_port=12355 \
    train.py
```

### DeepSpeed

```yaml
num_nodes: 4

resources:
  accelerators: A100:8

setup: |
  pip install deepspeed

run: |
  # Create hostfile
  echo "$SKYPILOT_NODE_IPS" | awk '{print $1 " slots=8"}' > /tmp/hostfile

  deepspeed --hostfile=/tmp/hostfile \
    --num_nodes=$SKYPILOT_NUM_NODES \
    --num_gpus=$SKYPILOT_NUM_GPUS_PER_NODE \
    train.py --deepspeed ds_config.json
```

### Ray Train

```yaml
num_nodes: 4

resources:
  accelerators: A100:8

run: |
  # Head node starts Ray head
  if [ "${SKYPILOT_NODE_RANK}" == "0" ]; then
    ray start --head --port=6379
    # Wait for workers
    sleep 30
    python train_ray.py
  else
    ray start --address=$(echo "$SKYPILOT_NODE_IPS" | head -n1):6379
  fi
```

## Sky Serve Advanced

### Multi-replica serving

```yaml
service:
  readiness_probe:
    path: /health
    initial_delay_seconds: 60
    period_seconds: 10

  replica_policy:
    min_replicas: 2
    max_replicas: 20
    target_qps_per_replica: 5.0
    upscale_delay_seconds: 60
    downscale_delay_seconds: 300

  load_balancing_policy: round_robin  # or least_connections
```

### Blue-green deployment

```bash
# Deploy new version
sky serve up -n my-service-v2 service_v2.yaml

# Test new version
curl https://my-service-v2.skypilot.cloud/health

# Switch traffic (update DNS/load balancer)
# Then terminate old version
sky serve down my-service-v1
```

### Service with multiple accelerator options

```yaml
service:
  replica_policy:
    min_replicas: 1
    max_replicas: 5

resources:
  accelerators:
    L40S: 1
    A100: 1
    A10G: 1
  any_of:
    - cloud: aws
    - cloud: gcp
```

## Cost Optimization

### Spot instance strategies

```yaml
resources:
  accelerators: A100:8
  use_spot: true
  spot_recovery: FAILOVER  # or FAILOVER_NO_WAIT

# Always checkpoint for spot jobs
file_mounts:
  /checkpoints:
    name: spot-checkpoints
    store: s3
    mode: MOUNT_CACHED
```

### Reserved instance hints

```yaml
resources:
  accelerators: A100:8
  # SkyPilot considers reserved instances in cost calculation
```

### Budget constraints

```bash
# Dry run to see cost estimate
sky launch task.yaml --dryrun

# Set max cluster cost (future feature)
# sky launch task.yaml --max-cost-per-hour 50
```

## Kubernetes Integration

### Using existing clusters

```bash
# Configure kubeconfig
export KUBECONFIG=~/.kube/config

# Verify
sky check kubernetes
```

### Pod configuration

```yaml
resources:
  cloud: kubernetes
  accelerators: A100:1

config:
  kubernetes:
    pod_config:
      spec:
        runtimeClassName: nvidia
        tolerations:
          - key: "nvidia.com/gpu"
            operator: "Exists"
            effect: "NoSchedule"
```

### Multi-cluster

```yaml
resources:
  any_of:
    - cloud: kubernetes
      infra: cluster1
    - cloud: kubernetes
      infra: cluster2
    - cloud: aws
```

## API Server Deployment

### Team setup

```bash
# Start API server
sky api serve --host 0.0.0.0 --port 8000

# Connect clients
sky api login --endpoint https://your-server:8000
```

### Authentication

```bash
# Create service account
sky api create-service-account my-service

# Use token in CI/CD
export SKYPILOT_API_TOKEN=...
sky launch task.yaml
```

## Advanced CLI Patterns

### Parallel cluster operations

```bash
# Launch multiple clusters in parallel
for i in {1..10}; do
  sky launch -c cluster-$i task.yaml --detach &
done
wait
```

### Batch job submission

```bash
# Submit many jobs
for config in configs/*.yaml; do
  name=$(basename $config .yaml)
  sky jobs launch -n $name $config
done

# Monitor all jobs
sky jobs queue
```

### Conditional execution

```yaml
run: |
  # Only run on head node
  if [ "${SKYPILOT_NODE_RANK}" == "0" ]; then
    python main.py
  else
    python worker.py
  fi
```

## Environment Management

### Environment variables

```yaml
envs:
  WANDB_PROJECT: my-project
  HF_TOKEN: $HF_TOKEN  # Inherit from local
  CUDA_VISIBLE_DEVICES: "0,1,2,3"

# Secrets (hidden in logs)
secrets:
  - WANDB_API_KEY
  - HF_TOKEN
```

### Config overrides

```yaml
config:
  # Override global config
  jobs:
    controller:
      resources:
        memory: 32
```

## Monitoring and Observability

### Log streaming

```bash
# Stream logs
sky logs mycluster

# Follow specific job
sky logs mycluster 1

# Managed job logs
sky jobs logs my-job --follow
```

### Integration with W&B/MLflow

```yaml
envs:
  WANDB_API_KEY: $WANDB_API_KEY
  WANDB_PROJECT: my-project

run: |
  wandb login $WANDB_API_KEY
  python train.py --wandb
```

## Debugging

### SSH access

```bash
# SSH to head node
ssh mycluster

# SSH to worker node
ssh mycluster-worker1

# Port forwarding
ssh -L 8080:localhost:8080 mycluster
```

### Interactive debugging

```bash
# Launch interactive cluster
sky launch -c debug --gpus A100:1

# SSH and debug
ssh debug
```

### Job inspection

```bash
# View job queue
sky queue mycluster

# Cancel specific job
sky cancel mycluster 1

# View job details
sky logs mycluster 1
```
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall