Skip to content

ericrisco/rsc-harness / skills/forecasting

forecasting

Verified

Use when projecting history forward — sales, demand, units, revenue, signups, traffic — into a defensible number with an error band: method by data shape, rolling-origin backtest, MASE vs the naive baseline. NOT an assumption-driven P&L or runway model (that is `financial-model`), NOT sizing reorder points or safety stock (that is `inventory`).

FileKindSize
SKILL.mdskill md10.8 KB
evals/README.mdmarkdown874 B
evals/cases.yamlconfig2.8 KB
references/accuracy-and-backtesting.mdmarkdown3.9 KB
references/methods-cheatsheet.mdmarkdown3.3 KB
scripts/verify.shexecutablescript3.3 KB

references/accuracy-and-backtesting.md

View on GitHub
# Accuracy and backtesting

The metrics decide what you ship. This is the reference for the formulas, the
rolling-origin mechanics, and how to read a backtest table.

## Metrics

### WAPE (a.k.a. WMAPE) — the default magnitude metric

```
WAPE = sum(|actual - forecast|) / sum(|actual|)
```

Weights error by volume. Stable when actuals approach zero, which is exactly where
MAPE fails. Report as a percentage. This is your headline accuracy number.

### MAPE — avoid for demand

```
MAPE = mean(|actual - forecast| / |actual|)
```

Divides by each actual, so a single near-zero period sends it to infinity. Fine for
smooth high-volume aggregate series; wrong for SKU-level and intermittent demand.
Default to WAPE and only use MAPE when actuals are reliably far from zero.

### Bias — direction of error

```
bias = sum(forecast - actual) / sum(|actual|)
```

Positive = systematic over-forecast, negative = under-forecast. Always report
alongside WAPE: WAPE is magnitude, bias is direction. A persistent bias is a
fixable, structural problem (wrong seasonality, ignored trend); random error around
zero bias is not.

### MASE — the pass/fail line vs naive

```
MASE = MAE(forecast) / MAE(naive forecast on the training set)
```

Scale-free. **MASE < 1.0 means you beat the naive forecast.** MASE ≥ 1.0 means the
naive forecast was more accurate — ship the naive one. For seasonal series the
denominator is the seasonal-naive MAE. This is the single number that justifies the
existence of your model.

### Interval quality — coverage and pinball

A point metric cannot grade an interval. Use:

- **Coverage:** the fraction of actuals that fall inside the interval. An 80%
  interval should contain ~80% of actuals in the backtest. Far below = overconfident
  (intervals too tight); far above = useless (too wide).
- **Pinball loss (quantile loss):** the proper scoring rule for a quantile forecast.
  Lower is better; it penalizes both miscalibration and width.

## Rolling-origin cross-validation

Never trust a single holdout — it is one sample of out-of-sample error and can be
lucky or unlucky. Rolling-origin (expanding-window) CV repeats the split across
several cutoffs and averages the error.

Mechanics (statsforecast `cross_validation`):

```
window 1:  train [.................]  test [###]
window 2:  train [....................]  test [###]
window 3:  train [.......................]  test [###]
```

- `h` — forecast horizon per window (match your real horizon).
- `n_windows` — how many cutoffs (≥ 3; more is better if history allows).
- `step_size` — gap between cutoffs (commonly `= h` for non-overlapping tests).

```python
cv = sf.cross_validation(df=df, h=12, n_windows=3, step_size=12)
# cv has: ds, cutoff, y, and one column per model
```

## Reading a backtest table

`cross_validation` returns the actual `y` and each model's prediction per cutoff.
Collapse to one row per model:

| model         | WAPE | bias  | MASE | verdict                         |
|---------------|------|-------|------|---------------------------------|
| SeasonalNaive | 0.18 | +0.02 | 1.00 | the bar (MASE is 1 by definition)|
| AutoETS       | 0.12 | -0.01 | 0.67 | beats naive — ship this         |
| AutoARIMA     | 0.13 | +0.06 | 0.72 | beats naive but biased high      |

Pick the lowest WAPE among models with **MASE < 1**. If no model has MASE < 1, ship
SeasonalNaive and report it honestly. Prefer a slightly higher-WAPE model with near-
zero bias over a lower-WAPE model with strong bias — bias compounds when you sum or
roll forward.

## Interval calibration

After choosing a model, check the chosen interval's coverage in the backtest. If an
80% interval covers only ~55% of actuals, the model is overconfident — widen the
level, switch to a method whose intervals are better calibrated for the series, or
state the limitation explicitly in the readout. An interval nobody calibrated is no
better than the point alone.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall