borghei/Claude-Skills / engineering/senior-ml-engineer
senior-ml-engineer
VerifiedProductionize ML models and LLM features: deployment specs, drift monitoring, RAG design and cost control. Use when deploying a model, building RAG, or monitoring drift.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 10.9 KB |
| assets/deployment_spec.json | config | 543 B |
| assets/monitoring_baseline.csv | other | 14.5 KB |
| assets/monitoring_current.csv | other | 14.6 KB |
| assets/rag_spec.json | config | 399 B |
| references/llm_integration_guide.md | markdown | 9.2 KB |
| references/mlops_production_patterns.md | markdown | 7.1 KB |
| references/production-ml-workflows.md | markdown | 8.5 KB |
| references/rag_system_architecture.md | markdown | 10.0 KB |
| references/tool-reference.md | markdown | 16.3 KB |
| scripts/ml_monitoring_suite.pyexecutable | script | 20.4 KB |
| scripts/model_deployment_pipeline.pyexecutable | script | 19.0 KB |
| scripts/rag_system_builder.pyexecutable | script | 18.2 KB |
references/production-ml-workflows.md
View on GitHub# Production ML Workflows
Read this when deploying a model, building an MLOps pipeline, integrating an LLM, implementing RAG, or setting up model monitoring — it holds the step-by-step workflows, code templates, comparison tables, troubleshooting, and success criteria.
## Model Deployment Workflow
Deploy a trained model to production with monitoring:
1. Export model to standardized format (ONNX, TorchScript, SavedModel)
2. Package model with dependencies in Docker container
3. Deploy to staging environment
4. Run integration tests against staging
5. Deploy canary (5% traffic) to production
6. Monitor latency and error rates for 1 hour
7. Promote to full production if metrics pass
8. **Validation:** p95 latency < 100ms, error rate < 0.1%
### Container Template
```dockerfile
FROM python:3.11-slim
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY model/ /app/model/
COPY src/ /app/src/
HEALTHCHECK CMD curl -f http://localhost:8080/health || exit 1
EXPOSE 8080
CMD ["uvicorn", "src.server:app", "--host", "0.0.0.0", "--port", "8080"]
```
### Serving Options
| Option | Latency | Throughput | Use Case |
|--------|---------|------------|----------|
| FastAPI + Uvicorn | Low | Medium | REST APIs, small models |
| Triton Inference Server | Very Low | Very High | GPU inference, batching |
| TensorFlow Serving | Low | High | TensorFlow models |
| TorchServe | Low | High | PyTorch models |
| Ray Serve | Medium | High | Complex pipelines, multi-model |
---
## MLOps Pipeline Setup
Establish automated training and deployment:
1. Configure feature store (Feast, Tecton) for training data
2. Set up experiment tracking (MLflow, Weights & Biases)
3. Create training pipeline with hyperparameter logging
4. Register model in model registry with version metadata
5. Configure staging deployment triggered by registry events
6. Set up A/B testing infrastructure for model comparison
7. Enable drift monitoring with alerting
8. **Validation:** New models automatically evaluated against baseline
### Feature Store Pattern
```python
from feast import Entity, Feature, FeatureView, FileSource
user = Entity(name="user_id", value_type=ValueType.INT64)
user_features = FeatureView(
name="user_features",
entities=["user_id"],
ttl=timedelta(days=1),
features=[
Feature(name="purchase_count_30d", dtype=ValueType.INT64),
Feature(name="avg_order_value", dtype=ValueType.FLOAT),
],
online=True,
source=FileSource(path="data/user_features.parquet"),
)
```
### Retraining Triggers
| Trigger | Detection | Action |
|---------|-----------|--------|
| Scheduled | Cron (weekly/monthly) | Full retrain |
| Performance drop | Accuracy < threshold | Immediate retrain |
| Data drift | PSI > 0.2 | Evaluate, then retrain |
| New data volume | X new samples | Incremental update |
---
## LLM Integration Workflow
Integrate LLM APIs into production applications:
1. Create provider abstraction layer for vendor flexibility
2. Implement retry logic with exponential backoff
3. Configure fallback to secondary provider
4. Set up token counting and context truncation
5. Add response caching for repeated queries
6. Implement cost tracking per request
7. Add structured output validation with Pydantic
8. **Validation:** Response parses correctly, cost within budget
### Provider Abstraction
```python
from abc import ABC, abstractmethod
from tenacity import retry, stop_after_attempt, wait_exponential
class LLMProvider(ABC):
@abstractmethod
def complete(self, prompt: str, **kwargs) -> str:
pass
@retry(stop=stop_after_attempt(3), wait=wait_exponential(min=1, max=10))
def call_llm_with_retry(provider: LLMProvider, prompt: str) -> str:
return provider.complete(prompt)
```
### Cost Management
Keep per-model input, output and cached-input prices in configuration with an as-of date,
copied from each provider's pricing page; they change several times a year. Track spend per
feature from logged token counts, route easy requests to the provider's small model, and
cache stable prompt prefixes.
---
## RAG System Implementation
Build retrieval-augmented generation pipeline:
1. Choose vector database (Pinecone, Qdrant, Weaviate)
2. Select embedding model based on quality/cost tradeoff
3. Implement document chunking strategy
4. Create ingestion pipeline with metadata extraction
5. Build retrieval with query embedding
6. Add reranking for relevance improvement
7. Format context and send to LLM
8. **Validation:** Response references retrieved context, no hallucinations
### Vector Database Selection
| Database | Hosting | Scale | Latency | Best For |
|----------|---------|-------|---------|----------|
| Pinecone | Managed | High | Low | Production, managed |
| Qdrant | Both | High | Very Low | Performance-critical |
| Weaviate | Both | High | Low | Hybrid search |
| Chroma | Self-hosted | Medium | Low | Prototyping |
| pgvector | Self-hosted | Medium | Medium | Existing Postgres |
### Chunking Strategies
| Strategy | Chunk Size | Overlap | Best For |
|----------|------------|---------|----------|
| Fixed | 500-1000 tokens | 50-100 | General text |
| Sentence | 3-5 sentences | 1 sentence | Structured text |
| Semantic | Variable | Based on meaning | Research papers |
| Recursive | Hierarchical | Parent-child | Long documents |
---
## Model Monitoring
Monitor production models for drift and degradation:
1. Set up latency tracking (p50, p95, p99)
2. Configure error rate alerting
3. Implement input data drift detection
4. Track prediction distribution shifts
5. Log ground truth when available
6. Compare model versions with A/B metrics
7. Set up automated retraining triggers
8. **Validation:** Alerts fire before user-visible degradation
### Drift Detection
```python
from scipy.stats import ks_2samp
def detect_drift(reference, current, threshold=0.05):
statistic, p_value = ks_2samp(reference, current)
return {
"drift_detected": p_value < threshold,
"ks_statistic": statistic,
"p_value": p_value
}
```
### Alert Thresholds
| Metric | Warning | Critical |
|--------|---------|----------|
| p95 latency | > 100ms | > 200ms |
| Error rate | > 0.1% | > 1% |
| PSI (drift) | > 0.1 | > 0.2 |
| Accuracy drop | > 2% | > 5% |
---
## Tech Stack
| Category | Tools |
|----------|-------|
| ML Frameworks | PyTorch, TensorFlow, Scikit-learn, XGBoost |
| LLM Frameworks | LangChain, LlamaIndex, DSPy |
| MLOps | MLflow, Weights & Biases, Kubeflow |
| Data | Spark, Airflow, dbt, Kafka |
| Deployment | Docker, Kubernetes, Triton |
| Databases | PostgreSQL, BigQuery, Pinecone, Redis |
---
## Troubleshooting
| Problem | Cause | Solution |
|---------|-------|----------|
| Model latency spikes after deployment | Container resource limits too low or cold starts on serverless | Pre-warm instances, increase CPU/memory limits, enable GPU request batching |
| Data drift alerts firing constantly | Reference distribution outdated or threshold too sensitive | Recalibrate reference window to recent 30 days, raise PSI warning threshold to 0.15 |
| Feature store serving stale features | TTL misconfigured or materialization job failing silently | Verify TTL matches data freshness SLA, add alerting on materialization job status |
| RAG retrieval returns irrelevant chunks | Chunk size too large or embedding model mismatch | Reduce chunk size to 300-500 tokens, switch to domain-tuned embedding model, add reranker |
| LLM provider rate limits hit in production | No request queuing or burst traffic exceeds quota | Implement token bucket rate limiter, add request queue with backpressure, configure fallback provider |
| Model accuracy degrades gradually | Concept drift in underlying data distribution | Enable automated retraining triggers on accuracy drop > 2%, schedule weekly evaluation jobs |
| A/B test results inconclusive after weeks | Insufficient traffic split or high-variance metric chosen | Increase treatment allocation to 10-20%, switch to lower-variance proxy metric, extend test duration |
---
## Success Criteria
- Model serving latency p99 under 100ms for real-time inference endpoints
- Zero data drift alerts unresolved for more than 48 hours
- Automated retraining pipeline triggers within 1 hour of performance threshold breach
- RAG system retrieval accuracy (hit rate at k=5) above 90% on evaluation set
- LLM integration uptime at 99.9% with provider fallback activating in under 2 seconds
- Feature store materialization freshness within defined TTL for all online features
- Model deployment rollback completes in under 5 minutes with zero dropped requests
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |