borghei/Claude-Skills / engineering/devops-workflow-engineer
devops-workflow-engineer
FlaggedGenerate and optimize GitHub Actions CI/CD workflows. Use when designing workflows, planning multi-environment deployments, optimizing pipeline cost and runtime, or implementing blue-green, canary, or rolling deployment strategies.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.8 KB |
| assets/cd-template.yml | config | 12.9 KB |
| assets/ci-template.yml | config | 9.0 KB |
| assets/sample_flawed_workflow.yml | config | 553 B |
| references/agentic-workflows-guide.md | markdown | 16.7 KB |
| references/deployment-strategies.md | markdown | 18.1 KB |
| references/github-actions-patterns.md | markdown | 18.1 KB |
| references/workflows-and-optimization.md | markdown | 6.2 KB |
| scripts/deployment_planner.py | script | 26.4 KB |
| scripts/pipeline_analyzer.py | script | 23.3 KB |
| scripts/workflow_generator.py | script | 25.9 KB |
references/deployment-strategies.md
View on GitHub# Deployment Strategies
Comprehensive guide to deployment strategies for production systems. Covers blue-green, canary, rolling, A/B testing, feature flags, and database migration handling during deployments.
---
## Table of Contents
1. [Strategy Comparison](#strategy-comparison)
2. [Blue-Green Deployment](#blue-green-deployment)
3. [Canary Deployment](#canary-deployment)
4. [Rolling Deployment](#rolling-deployment)
5. [A/B Testing Deployments](#ab-testing-deployments)
6. [Feature Flags](#feature-flags)
7. [Database Migrations During Deploy](#database-migrations-during-deploy)
8. [Choosing the Right Strategy](#choosing-the-right-strategy)
---
## Strategy Comparison
| Aspect | Blue-Green | Canary | Rolling | A/B Testing |
|--------|-----------|--------|---------|-------------|
| Zero downtime | Yes | Yes | Yes | Yes |
| Rollback speed | Instant | Instant | Minutes | Instant |
| Infrastructure cost | 2x during deploy | 1.1-1.5x | 1x | 1.1-1.5x |
| Complexity | Low | High | Low | High |
| Production testing | Full env before switch | Gradual real traffic | Mixed versions | Segment-based |
| Best for | Critical apps | High-traffic services | Stateless services | UX experiments |
| Risk level | Low | Very low | Medium | Very low |
---
## Blue-Green Deployment
### Overview
Blue-green deployment maintains two identical production environments. At any time, only one (say "blue") serves live traffic. New versions are deployed to the inactive environment ("green"), validated, and then traffic is switched.
### Architecture
```
Load Balancer
/ \
/ \
[Blue - v1.2] [Green - v1.3]
(ACTIVE) (STAGING)
\ /
\ /
Shared Database
```
### Implementation Steps
1. **Deploy to green environment**
```bash
# Deploy new version to green
kubectl apply -f deployment-green.yaml
kubectl rollout status deployment/app-green --timeout=300s
```
2. **Run validation on green**
```bash
# Health checks
curl -sf https://green.internal.example.com/health || exit 1
# Smoke tests against green
SMOKE_URL=https://green.internal.example.com pytest tests/smoke/
```
3. **Switch traffic**
```bash
# Update load balancer to point to green
# AWS ALB example:
aws elbv2 modify-listener --listener-arn $LISTENER_ARN \
--default-actions Type=forward,TargetGroupArn=$GREEN_TG_ARN
# Or Kubernetes service selector:
kubectl patch service app-service -p '{"spec":{"selector":{"version":"green"}}}'
```
4. **Monitor**
- Watch error rates for 15-30 minutes.
- Compare latency baselines.
- Verify business metrics (conversion, API success rate).
5. **Finalize or rollback**
```bash
# If healthy: decommission blue (or keep as rollback)
# If unhealthy: switch back to blue
kubectl patch service app-service -p '{"spec":{"selector":{"version":"blue"}}}'
```
### GitHub Actions Integration
```yaml
jobs:
deploy-green:
environment: production-green
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- name: Deploy to green
run: ./scripts/deploy.sh green ${{ github.sha }}
- name: Validate green
run: ./scripts/validate.sh green
switch-traffic:
needs: deploy-green
environment: production
runs-on: ubuntu-latest
steps:
- name: Switch traffic to green
run: ./scripts/switch-traffic.sh green
- name: Monitor (5 min)
run: ./scripts/monitor.sh --duration 300 --threshold 0.01
```
### When to Use
- Applications where instant rollback is critical.
- Systems that can afford 2x infrastructure during deployments.
- When you need to validate the full environment before exposing to users.
### When to Avoid
- Very large infrastructure (cost doubles during deploy).
- Stateful services with data that diverges between environments.
- When database schema changes require both versions to coexist.
---
## Canary Deployment
### Overview
Canary deployment routes a small fraction of traffic to the new version while the majority continues on the stable version. Traffic percentage increases gradually based on success metrics.
### Architecture
```
Load Balancer / Service Mesh
/ | \
95% | 5%
/ | \
[Stable v1.2] | [Canary v1.3]
(10 replicas) | (1 replica)
|
Metric Collection
(errors, latency)
```
### Traffic Split Schedule
| Phase | Canary % | Duration | Promotion Gate |
|-------|---------|----------|----------------|
| Deploy | 0% | 5 min | Canary pods healthy |
| Phase 1 | 5% | 15 min | Error rate < 0.1%, P99 < 200ms |
| Phase 2 | 25% | 30 min | Error rate < 0.1%, P99 < 200ms |
| Phase 3 | 50% | 60 min | All metrics within baseline |
| Phase 4 | 100% | -- | Full promotion |
### Implementation with Kubernetes
Using Argo Rollouts:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: my-app
spec:
replicas: 10
strategy:
canary:
steps:
- setWeight: 5
- pause: { duration: 15m }
- setWeight: 25
- pause: { duration: 30m }
- setWeight: 50
- pause: { duration: 60m }
- setWeight: 100
canaryMetadata:
labels:
role: canary
stableMetadata:
labels:
role: stable
analysis:
templates:
- templateName: success-rate
startingStep: 1
args:
- name: service-name
value: my-app
```
### Analysis Template
```yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: success-rate
spec:
args:
- name: service-name
metrics:
- name: success-rate
interval: 1m
successCondition: result[0] >= 0.99
failureCondition: result[0] < 0.95
failureLimit: 3
provider:
prometheus:
address: http://prometheus:9090
query: |
sum(rate(http_requests_total{service="{{args.service-name}}",status!~"5.."}[5m]))
/
sum(rate(http_requests_total{service="{{args.service-name}}"}[5m]))
```
### Implementation with Istio
```yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: my-app
spec:
hosts:
- my-app
http:
- route:
- destination:
host: my-app
subset: stable
weight: 95
- destination:
host: my-app
subset: canary
weight: 5
```
### When to Use
- High-traffic services where a bad deploy affects many users.
- When you have good observability and can detect issues at low traffic percentages.
- Services where you want real production validation before full rollout.
### When to Avoid
- Low-traffic services (not enough signal at 5%).
- Services without good metrics/monitoring.
- When speed of deployment is more important than safety.
---
## Rolling Deployment
### Overview
Rolling deployment replaces instances of the old version with the new version one at a time (or in configured batch sizes). At least some instances are always available.
### Architecture
```
Time T0: [v1] [v1] [v1] [v1] (all old)
Time T1: [v2] [v1] [v1] [v1] (1 updated)
Time T2: [v2] [v2] [v1] [v1] (2 updated)
Time T3: [v2] [v2] [v2] [v1] (3 updated)
Time T4: [v2] [v2] [v2] [v2] (all new)
```
### Kubernetes Configuration
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 4
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # Max extra pods during update
maxUnavailable: 0 # Always maintain full capacity
template:
spec:
containers:
- name: app
image: my-app:v2
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 15
periodSeconds: 10
```
### Key Parameters
| Parameter | Description | Recommended Default |
|-----------|-------------|-------------------|
| `maxSurge` | Extra pods allowed during update | 25% or 1 |
| `maxUnavailable` | Pods that can be unavailable | 0 (safest) or 25% |
| `minReadySeconds` | Wait after ready before next | 10-30 seconds |
| `progressDeadlineSeconds` | Timeout for rollout | 600 seconds |
### Rollback
```bash
# Automatic rollback on failure
kubectl rollout undo deployment/my-app
# Rollback to specific revision
kubectl rollout undo deployment/my-app --to-revision=3
# Check rollout history
kubectl rollout history deployment/my-app
```
### When to Use
- Stateless services with multiple replicas.
- Kubernetes-native deployments.
- When infrastructure cost must stay flat (no extra environment).
### When to Avoid
- When both versions cannot serve traffic simultaneously (breaking API changes).
- Single-instance deployments (no redundancy during update).
- When instant rollback is required (rolling back is another rolling update).
---
## A/B Testing Deployments
### Overview
A/B testing deployments route traffic to different versions based on user attributes (not random traffic splitting). This enables data-driven decisions about which version performs better.
### Difference from Canary
| Aspect | Canary | A/B Test |
|--------|--------|----------|
| Goal | Risk mitigation | Feature validation |
| Routing | Random percentage | User attributes |
| Duration | Hours | Days to weeks |
| Metrics | Error rate, latency | Business KPIs |
| Rollback trigger | Technical failure | Experiment conclusion |
### Routing Strategies
**Header-based:**
```yaml
# Istio VirtualService
http:
- match:
- headers:
x-experiment-group:
exact: "variant-b"
route:
- destination:
host: my-app
subset: variant-b
- route:
- destination:
host: my-app
subset: variant-a
```
**Cookie-based:**
```yaml
http:
- match:
- headers:
cookie:
regex: ".*ab_group=B.*"
route:
- destination:
host: my-app
subset: variant-b
```
**User-segment based (application level):**
```python
def get_variant(user):
"""Determine which variant to show based on user attributes."""
if user.id % 100 < 50:
return "A"
return "B"
```
### Metrics to Track
- **Primary metric:** Conversion rate, revenue per user, engagement time.
- **Guardrail metrics:** Error rate, latency, bounce rate (must not degrade).
- **Statistical significance:** Typically need p < 0.05 with adequate sample size.
### When to Use
- Validating new UI/UX against existing.
- Testing pricing changes.
- Comparing different recommendation algorithms.
---
## Feature Flags
### Overview
Feature flags decouple deployment from release. Code is deployed but features are toggled on/off independently, enabling gradual rollout, instant kill switches, and A/B testing without redeployment.
### Flag Types
| Type | Lifespan | Example |
|------|----------|---------|
| Release flag | Days to weeks | New checkout flow |
| Experiment flag | Weeks to months | Pricing experiment |
| Ops flag | Permanent | Maintenance mode, rate limits |
| Permission flag | Permanent | Premium features, beta access |
### Implementation Patterns
**Simple boolean flag:**
```python
def checkout(request):
if flags.is_enabled("new-checkout-v2"):
return new_checkout(request)
return legacy_checkout(request)
```
**Percentage rollout:**
```python
def checkout(request):
if flags.is_enabled("new-checkout-v2", percentage=10):
return new_checkout(request)
return legacy_checkout(request)
```
**User-targeted flag:**
```python
def checkout(request):
if flags.is_enabled("new-checkout-v2", user=request.user,
rules={"plan": "premium", "country": "US"}):
return new_checkout(request)
return legacy_checkout(request)
```
### Flag Lifecycle
```
1. Create flag (disabled) -----> Code deploys with flag check
2. Enable for internal users --> Test in production
3. Enable for 5% of users ----> Monitor metrics
4. Increase to 25%, 50% ------> Gradual rollout
5. Enable for 100% -----------> Full release
6. Remove flag from code ------> Clean up (critical step!)
```
### Best Practices
- **Name flags clearly:** `enable-new-checkout-flow` not `flag-123`.
- **Set expiration dates:** Stale flags are tech debt.
- **Log flag evaluations:** Know which users see which variant.
- **Have a kill switch process:** Document how to disable a flag in an emergency.
- **Clean up after rollout:** Remove flag code once fully launched.
- **Limit active flags:** More than 10-15 active flags creates combinatorial complexity.
### Feature Flag in GitHub Actions
```yaml
jobs:
deploy:
steps:
- name: Set feature flags for environment
run: |
# Using a hypothetical flag management CLI
flag-ctl set new-checkout-v2 \
--env ${{ inputs.environment }} \
--percentage 5 \
--description "Gradual rollout of new checkout"
```
### Services
| Service | Type | Key Feature |
|---------|------|-------------|
| LaunchDarkly | SaaS | Real-time flag evaluation, experimentation |
| Unleash | Open source | Self-hosted, Kubernetes-native |
| Flagsmith | Open source + SaaS | Remote config + flags |
| Split.io | SaaS | Feature flags + experimentation platform |
| Custom | DIY | Simple JSON/YAML config file |
---
## Database Migrations During Deploy
### The Challenge
Database schema changes during deployment create risk because:
- Old code may not work with new schema.
- New code may not work with old schema.
- Both versions run simultaneously during rolling/canary/blue-green deployments.
### Golden Rule: Backward-Compatible Migrations
Every migration must be compatible with **both** the current and previous version of the application code.
### Safe Migration Patterns
**1. Add a column (safe)**
```sql
-- Migration: Add column with default
ALTER TABLE users ADD COLUMN email_verified BOOLEAN DEFAULT FALSE;
-- Old code ignores the new column. New code uses it.
```
**2. Remove a column (two-phase)**
```
Phase 1 (deploy v2): Stop reading the column in code. Deploy.
Phase 2 (deploy v3): Drop the column in migration. Deploy.
```
**3. Rename a column (three-phase)**
```
Phase 1: Add new column, write to both old and new.
Phase 2: Migrate data, read from new column.
Phase 3: Drop old column.
```
**4. Add an index (safe, but watch performance)**
```sql
-- Use CONCURRENTLY to avoid locking the table
CREATE INDEX CONCURRENTLY idx_users_email ON users(email);
```
**5. Change column type (two-phase)**
```
Phase 1: Add new column with new type, dual-write.
Phase 2: Migrate data, switch reads to new column.
Phase 3: Drop old column.
```
### Unsafe Migration Patterns (Avoid)
| Migration | Risk | Safe Alternative |
|-----------|------|-----------------|
| DROP COLUMN | Old code breaks | Two-phase: stop using, then drop |
| RENAME COLUMN | Both versions break | Three-phase: add, migrate, drop |
| ALTER TYPE (in-place) | Table lock, old code breaks | Add new column, migrate |
| NOT NULL without default | INSERT fails for old code | Add with DEFAULT first |
| DROP TABLE | Everything breaks | Deprecate, stop references, then drop |
### Migration Execution in CI/CD
```yaml
jobs:
migrate:
runs-on: ubuntu-latest
environment: ${{ inputs.environment }}
steps:
- uses: actions/checkout@v7
- name: Backup database
run: ./scripts/db-backup.sh ${{ inputs.environment }}
- name: Run migration (dry run)
run: ./scripts/db-migrate.sh --dry-run --env ${{ inputs.environment }}
- name: Run migration
run: ./scripts/db-migrate.sh --env ${{ inputs.environment }}
- name: Verify migration
run: ./scripts/db-verify.sh --env ${{ inputs.environment }}
deploy:
needs: migrate
# ... deploy application code
```
### Migration Rollback
- Always write a corresponding `down` migration.
- Test rollback in staging before deploying to production.
- If a migration is not reversible, document it and plan accordingly.
```python
# Example: Django migration with reverse
class Migration(migrations.Migration):
operations = [
migrations.AddField(
model_name='user',
name='email_verified',
field=models.BooleanField(default=False),
),
]
# Django auto-generates reverse: RemoveField
```
---
## Choosing the Right Strategy
### Decision Tree
```
Start
|
v
Is zero-downtime required?
|
No --> Simple deployment (stop old, start new)
|
Yes
|
v
Is instant rollback critical?
|
No --> Rolling deployment
|
Yes
|
v
Can you afford 2x infrastructure?
|
Yes --> Blue-green deployment
|
No
|
v
Do you have good observability (metrics, alerting)?
|
No --> Blue-green (simpler to manage)
|
Yes
|
v
Is traffic high enough for meaningful canary signal?
|
No --> Blue-green
|
Yes --> Canary deployment
```
### Strategy Selection by Service Type
| Service Type | Recommended Primary | Recommended Secondary |
|-------------|--------------------|-----------------------|
| User-facing web app | Blue-green | Canary |
| API service | Canary | Rolling |
| Background worker | Rolling | Blue-green |
| Stateful service | Blue-green | Rolling (careful) |
| Database | Rolling (with migrations) | N/A |
| Shared library | Version publish | N/A |
| Mobile app | Staged rollout (canary) | Feature flags |
| Infrastructure | Rolling (Terraform) | Blue-green |
### Combining Strategies
Production deployments often combine multiple strategies:
1. **Feature flags + Canary:** Deploy with feature hidden behind flag, canary the deployment, then gradually enable the flag.
2. **Blue-green + Database migration:** Run backward-compatible migration first, then blue-green switch the application.
3. **Rolling + Health checks:** Standard Kubernetes rolling update with readiness probes acting as automatic gates.
4. **Canary + A/B test:** Use canary for risk mitigation during deploy, then A/B test the feature over days.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |