Skip to content

DAAF-Contribution-Community/daaf / .claude/skills/tidymodels

tidymodels

Verified

R machine learning with tidymodels: recipes (preprocessing), parsnip (model specs), workflows (fit pipelines), tune (hyperparameters), rsample (resampling). Engines: ranger (RF), glmnet (lasso/ridge), xgboost, kknn. Model interpretation (permutation importance, PDP/ICE, SHAP via DALEX/iml/kernelshap) and fairness assessment (fairmodels). UMAP via uwot. Use when execution language is R. Python equivalent: scikit-learn. For econometric regression use fixest or r-stats.

FileKindSize
SKILL.mdskill md19.6 KB
references/engines.mdmarkdown9.3 KB
references/evaluation.mdmarkdown6.2 KB
references/fairness.mdmarkdown7.1 KB
references/gotchas.mdmarkdown9.6 KB
references/interpretation.mdmarkdown9.5 KB
references/models.mdmarkdown7.8 KB
references/quickstart.mdmarkdown5.4 KB
references/recipes.mdmarkdown7.2 KB
references/resampling.mdmarkdown7.2 KB
references/tuning.mdmarkdown7.4 KB
references/unsupervised.mdmarkdown6.3 KB
references/workflows.mdmarkdown5.2 KB

references/evaluation.md

View on GitHub
# Evaluation

Yardstick metrics for evaluating model performance: regression metrics (RMSE, MAE,
R-squared), classification metrics (accuracy, ROC-AUC, precision, recall, F1,
confusion matrix), metric sets, and integration with tuning results.

## Evaluation Pattern

In tidymodels, evaluation uses `augment()` to get predictions joined to truth,
then yardstick metric functions:

```r
library(tidymodels)

# Get predictions with truth
results <- fit |> augment(new_data = test_data)

# Compute metrics
rmse(results, truth = outcome, estimate = .pred)
accuracy(results, truth = outcome, estimate = .pred_class)
```

## Regression Metrics

```r
# --- RMSE (root mean squared error) ---
rmse(results, truth = y, estimate = .pred)

# --- MAE (mean absolute error) ---
mae(results, truth = y, estimate = .pred)

# --- R-squared ---
rsq(results, truth = y, estimate = .pred)

# --- MAPE (mean absolute percentage error) ---
mape(results, truth = y, estimate = .pred)

# --- Huber loss (robust to outliers) ---
huber_loss(results, truth = y, estimate = .pred)
```

| Metric | Range | Interpretation |
|--------|-------|---------------|
| `rmse` | [0, Inf) | Same units as target. Lower = better. |
| `mae` | [0, Inf) | Same units as target. Robust to outliers. Lower = better. |
| `rsq` | [0, 1] | Fraction of variance explained. Higher = better. |
| `mape` | [0, Inf) | Percentage error. Lower = better. |

## Classification Metrics

### Accuracy

```r
accuracy(results, truth = outcome, estimate = .pred_class)
```

Only meaningful when classes are balanced. For imbalanced data, use precision/
recall/F1 or ROC-AUC.

### ROC AUC

Requires probability predictions (`.pred_{level}` columns from `augment()`):

```r
# Binary classification
roc_auc(results, truth = outcome, .pred_positive_class)

# Multi-class
roc_auc(results, truth = outcome, .pred_class1, .pred_class2, .pred_class3)
# Or use the tidyselect helper:
roc_auc(results, truth = outcome, starts_with(".pred_"), -c(.pred_class))
```

### ROC Curve

```r
# Get ROC curve data for plotting
roc_data <- roc_curve(results, truth = outcome, .pred_positive_class)

# Plot
autoplot(roc_data)
```

### Precision, Recall, F1

```r
# Precision (positive predictive value)
precision(results, truth = outcome, estimate = .pred_class)

# Recall (sensitivity)
recall(results, truth = outcome, estimate = .pred_class)

# F1 (harmonic mean of precision and recall)
f_meas(results, truth = outcome, estimate = .pred_class)
```

For multi-class, specify the estimator:

```r
# Macro average (unweighted mean across classes)
f_meas(results, truth = outcome, estimate = .pred_class, estimator = "macro")

# Weighted average (weighted by class support)
f_meas(results, truth = outcome, estimate = .pred_class, estimator = "macro_weighted")
```

### Confusion Matrix

```r
# Confusion matrix
cm <- conf_mat(results, truth = outcome, estimate = .pred_class)
cm

# Visual display
autoplot(cm, type = "heatmap")

# Summary statistics from confusion matrix
summary(cm)
# Returns: accuracy, kappa, sensitivity, specificity, ppv, npv, etc.
```

### Sensitivity and Specificity

```r
sensitivity(results, truth = outcome, estimate = .pred_class)
specificity(results, truth = outcome, estimate = .pred_class)
```

## metric_set(): Multiple Metrics at Once

Bundle multiple metrics into a single function call:

```r
# --- Regression metric set ---
reg_metrics <- metric_set(rmse, mae, rsq)
reg_metrics(results, truth = y, estimate = .pred)
# Returns a tibble with .metric, .estimator, .estimate columns

# --- Classification metric set ---
class_metrics <- metric_set(accuracy, f_meas, precision, recall)
class_metrics(results, truth = outcome, estimate = .pred_class)
```

## Metrics with Tuning Results

When using `tune_grid()` or `fit_resamples()`, metrics are collected
automatically:

```r
# Default metrics
tune_results <- tune_grid(wf, resamples = folds, grid = 20)
collect_metrics(tune_results)

# Custom metrics
my_metrics <- metric_set(accuracy, roc_auc, f_meas)
tune_results <- tune_grid(wf, resamples = folds, grid = 20, metrics = my_metrics)
collect_metrics(tune_results)
```

Default metrics by mode:

| Mode | Default Metrics |
|------|----------------|
| regression | rmse, rsq |
| classification | accuracy, roc_auc (if probabilities available) |

## Comparing Models

To compare multiple models, collect predictions from each and compute metrics:

```r
# Fit multiple workflows
fit_rf <- wf_rf |> fit(data = train_data)
fit_glm <- wf_glm |> fit(data = train_data)
fit_xgb <- wf_xgb |> fit(data = train_data)

# Augment test data with each
results_rf <- fit_rf |> augment(new_data = test_data) |> mutate(model = "RF")
results_glm <- fit_glm |> augment(new_data = test_data) |> mutate(model = "GLM")
results_xgb <- fit_xgb |> augment(new_data = test_data) |> mutate(model = "XGBoost")

# Combine and compute metrics by model
all_results <- bind_rows(results_rf, results_glm, results_xgb)
all_results |>
  group_by(model) |>
  accuracy(truth = outcome, estimate = .pred_class)
```

## Event Level

For binary classification, yardstick needs to know which class is the "event"
(positive class). By default, yardstick treats the **first level** of the factor
as the event.

```r
# Check factor levels
levels(results$outcome)
# [1] "no"  "yes"  -- "no" is the event by default

# Specify event level explicitly
sensitivity(results, truth = outcome, estimate = .pred_class, event_level = "second")
roc_auc(results, truth = outcome, .pred_yes, event_level = "second")
```

## Quick Reference

| Task | Code |
|------|------|
| RMSE | `rmse(results, truth = y, estimate = .pred)` |
| MAE | `mae(results, truth = y, estimate = .pred)` |
| R-squared | `rsq(results, truth = y, estimate = .pred)` |
| Accuracy | `accuracy(results, truth = y, estimate = .pred_class)` |
| ROC AUC | `roc_auc(results, truth = y, .pred_positive)` |
| Precision | `precision(results, truth = y, estimate = .pred_class)` |
| Recall | `recall(results, truth = y, estimate = .pred_class)` |
| F1 | `f_meas(results, truth = y, estimate = .pred_class)` |
| Confusion matrix | `conf_mat(results, truth = y, estimate = .pred_class)` |
| Metric set | `metric_set(rmse, mae, rsq)` |
| ROC curve | `roc_curve(results, truth = y, .pred_positive) \|> autoplot()` |
| Event level | `metric(..., event_level = "second")` |
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall