artubss/SKILLS-CLAUDE-CODE / favorites/tpl-situacao-machine-learning-integracao
tpl-situacao-machine-learning-integracao
VerifiedTemplate do pack (situacao/16-machine-learning-integracao.md). Orienta o agente em tarefas situacionais como debug, seguranca e refactor alinhado a esse contexto.
SITUATION: Machine Learning Model Integration
Skill gerado a partir do pack templates-claude-code. Arquivo de origem: situacao/16-machine-learning-integracao.md. Use como baseline e adapte ao projeto antes de mudancas grandes.
Conteudo do template
CONTEXT
Use this configuration when integrating a trained machine learning model into a production application — whether it's a recommendation engine, fraud classifier, sentiment analyzer, content moderator, or any model served via API or embedded SDK. ML models in production have unique failure modes that differ from traditional software: they can be "right" statistically but wrong for individual cases, they degrade silently as data distributions shift, and they require careful A/B testing to safely update.
OBJECTIVES
- Serve model predictions reliably with graceful fallback when the model is unavailable
- Validate all inputs before inference to prevent nonsensical predictions
- Monitor for data drift and model performance degradation post-launch
- Enable safe A/B testing of model versions without code deploys
- Collect ground truth for continuous model improvement
APPROACH RULES
-
Input validation before inference is mandatory. Never pass raw user input to a model. Validate type, range, cardinality, and format. A model given garbage input produces confident garbage output — this is worse than an error.
-
Always have a fallback. When the model is unavailable (service down, timeout, error), fall back to a reasonable default: most popular items, rule-based logic, or graceful degradation (show without personalization). Never fail the entire request because the model is unavailable.
-
Log every prediction for ground truth collection. Log: input features, prediction output, confidence score, model version, and the user's eventual action. Without ground truth, you cannot measure model accuracy in production.
-
Model versions are deployment artifacts. Like Docker images, model versions must be immutable, tagged, and deployed explicitly. Never load "the latest model" automatically from a shared location in production.
-
A/B test every model change. Updating a model is a behavior change. It must go through the same A/B testing process as a product feature change. Traffic splitting, metric collection, and winner determination process must all apply.
-
Confidence thresholds, not just predictions. For high-stakes decisions (fraud, content moderation), set a confidence threshold below which the model defers to a human or applies a conservative default. A 51% confidence fraud prediction should not automatically block an account.
-
Feature drift is the #1 cause of production ML failures. Monitor input feature distributions. If the distribution of features being sent for inference drifts significantly from training distribution, model accuracy degrades silently.
ROUTING TABLE
| If you encounter | Then |
|---|---|
| Model service timeout | Activate fallback immediately. Log the timeout. Alert if timeout rate > 1%. |
| Model returns confidence < threshold | Apply conservative fallback logic. Do not act on low-confidence predictions for high-stakes decisions. |
| Input feature out of expected range | Reject or clip the value. Log as a data quality anomaly. Monitor frequency — could indicate upstream bug. |
| Model accuracy metrics degrading | Check: input feature drift, data pipeline issues, population shift. Trigger re-evaluation process. |
| New model version ready to deploy | Use A/B test with 5% → 20% → 50% → 100% rollout. Define success metric before starting. |
| Model serving latency increasing | Check: batch size, hardware utilization, model size. Consider: model quantization, caching, async serving. |
| Missing feature values | Apply agreed-upon imputation strategy (mean, median, mode, zero). Never send NULL to model. |
| Model predicts poorly on a subgroup | Evaluate dataset bias. Auditability required. Consult domain experts. Do not ship biased predictions for protected groups. |
| No ground truth collection | Build it before model goes live. Without ground truth, you cannot measure or improve. |
| Model file too large for web (frontend ML) | Quantize, prune, or use knowledge distillation. Or move to server-side inference. |
Model Serving Architecture
interface ModelPrediction {
result: string | number | boolean
confidence: number
modelVersion: string
latencyMs: number
}
class RecommendationService {
constructor(
private modelClient: ModelClient,
private fallback: FallbackRecommendations,
private logger: Logger,
private metrics: MetricsClient
) {}
async getRecommendations(userId: string, context: UserContext): Promise<Product[]> {
const start = Date.now()
try {
// 1. Validate and prepare features
const features = this.prepareFeatures(userId, context)
this.validateFeatures(features)
// 2. Call model with timeout
const prediction = await Promise.race([
this.modelClient.predict(features),
timeout(500, new Error('Model timeout'))
])
// 3. Apply confidence threshold
if (prediction.confidence < 0.6) {
this.logger.warn({ userId, confidence: prediction.confidence }, 'Low confidence prediction')
return this.fallback.getPopularItems(context.category)
}
// 4. Log prediction for ground truth
this.logPrediction(userId, features, prediction)
this.metrics.histogram('model.latency', Date.now() - start)
this.metrics.increment('model.predictions.served')
return prediction.result as Product[]
} catch (err) {
this.metrics.increment('model.predictions.fallback')
this.logger.error({ err, userId }, 'Model inference failed, using fallback')
return this.fallback.getPopularItems(context.category)
}
}
private logPrediction(userId: string, features: Features, prediction: ModelPrediction): void {
// Critical: store this for ground truth collection
this.predictionLog.save({
timestamp: new Date().toISOString(),
userId,
features,
prediction: prediction.result,
confidence: prediction.confidence,
modelVersion: prediction.modelVersion
})
}
}
Feature Drift Monitoring
from scipy import stats
import numpy as np
class FeatureDriftMonitor:
def __init__(self, reference_distribution: dict):
"""
reference_distribution: {feature_name: np.array of training values}
"""
self.reference = reference_distribution
def check_drift(self, current_window: dict, threshold: float = 0.05) -> dict:
"""
Uses KS test to detect distribution drift.
Returns: {feature_name: {'drifted': bool, 'p_value': float}}
"""
results = {}
for feature, current_values in current_window.items():
if feature not in self.reference:
continue
ks_stat, p_value = stats.ks_2samp(
self.reference[feature],
current_values
)
drifted = p_value < threshold
results[feature] = {'drifted': drifted, 'p_value': p_value, 'ks_stat': ks_stat}
if drifted:
logger.warning({'feature': feature, 'p_value': p_value}, 'Feature drift detected')
metrics.increment('ml.feature_drift', tags=[f'feature:{feature}'])
return results
DO NOT
- DO NOT deploy a new model version to 100% of traffic without A/B testing
- DO NOT make irreversible decisions (account banning, transaction blocking) without a human review step for borderline cases
- DO NOT skip input validation — garbage in produces confident garbage out
- DO NOT use a single aggregate accuracy metric — slice by user segments and time periods
- DO NOT ignore low-confidence predictions — they often signal distribution shift
- DO NOT load ML models synchronously in web request handlers — pre-load at startup
- DO NOT store raw PII as features in prediction logs — hash/anonymize user identifiers
- DO NOT serve a model that wasn't tested on data representative of production distribution
OUTPUT FORMAT
For each ML integration, deliver:
ML Integration Spec:
## ML Integration: [Model Name]
### Model
- Type: Recommendation / Classification / Regression
- Framework: scikit-learn 1.3 / PyTorch 2.1 / ONNX
- Version: v2.4.1 (immutable artifact SHA: abc123)
- Training data: transactions 2023-01 to 2024-01
### Serving
- Endpoint: https://ml-api.internal/v1/recommend
- Method: REST POST / gRPC / embedded
- SLA: P95 < 200ms
- Timeout: 500ms (then fallback)
### Input Features
| Feature | Type | Range/Values | Handling if Missing |
|---------|------|-------------|--------------------|
| user_age_days | int | 0-3650 | median imputation (365) |
| purchase_count | int | 0-999+ | clip at 999 |
| category | enum | [electronics, clothing, ...] | "unknown" category |
### Fallback Strategy
- Model unavailable: serve most popular items for the session's category
- Low confidence (< 0.6): serve curated editorial picks
- Feature validation failure: serve bestsellers
### Monitoring
- Latency tracking: P50, P95, P99 histograms
- Error rate: model unavailable, timeout, validation failure
- Feature drift: daily KS test vs training distribution
- Business metric: click-through rate on recommendations
### Ground Truth Collection
- Log: prediction request + features + output + model version
- Ground truth: user click/purchase on predicted item (join after 24h)
- Accuracy report: weekly, sliced by user segment and model version
QUALITY GATES
- Input validation implemented and tested with adversarial inputs
- Fallback path tested independently (works without model available)
- Model version pinned and immutable in production config
- Ground truth collection implemented before first prediction served
- Confidence threshold defined and applied for high-stakes predictions
- Feature drift monitoring in place with alert threshold
- P95 inference latency < SLA with model loaded (test before launch)
- A/B test framework in place before first model update
- No PII stored in raw form in prediction logs
- On-call runbook for: model unavailable, high error rate, detected drift
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |