khuynh22/agent-dev-team / skills/ml-lifecycle
ml-lifecycle
VerifiedML Lifecycle
A model in a notebook is a claim. A model in production with a baseline, a monitor, and a rollback is a system. Offline accuracy is not evidence of online value, and the first suspiciously good result is usually leakage rather than skill.
Standing reference: references/data-quality-checklist.md.
Process
-
Measure a non-learned baseline first. Last observed value, majority class, a three-line heuristic. Every subsequent number is a delta against it. If the model is within noise of the baseline, the correct deliverable is the baseline.
-
Split by time when the data has time in it, and hold out a period the model never sees. A random split on temporal data trains on the future and produces a number that does not survive next week.
-
Hunt leakage before celebrating. Any feature computed after the label exists, or read from a table that is updated retroactively, is leakage. Audit each feature by asking what its value would have been at prediction time. A first result far above the baseline is an alarm, not a win.
-
Compute features once. Training and serving read the same code path. Otherwise you ship a model that scores differently in production than in the notebook, and the investigation costs a week. If the paths must differ, measure the skew as a number.
-
Evaluate slices, not just the aggregate. By segment, cohort, volume tier, and on the tail. Report the worst slice in the headline, because the worst slice is what generates complaints and it is invisible in the average.
-
Tie the offline metric to a product metric. State the link explicitly: this ranking metric maps to that conversion outcome because of this mechanism. An offline win with no stated link is a number, not a reason to ship.
-
Roll out behind shadow, then canary. Score live traffic without acting on it, compare against the incumbent on the same requests, then take a percentage with an abort condition written down in advance.
-
Monitor inputs and outputs after launch. Feature distribution drift, prediction distribution drift, and the product metric. A degrading model raises no exception; the only signal is a monitor somebody built.
-
Write the rollback before the launch. Which version, how it is pinned, how long it takes, and what triggers it. Retraining is not a rollback: it takes hours and might not help.
Output
## Model: <name> v<n>
- **Baseline:** <what> — <metric> = <n>
- **Model:** <metric> = <n> (delta <n>), worst slice <name> = <n> (n=<n>)
- **Split:** temporal, train <range>, holdout <range>
- **Leakage audit:** <features checked, what was removed and why>
- **Train/serve parity:** <shared path | measured skew>
- **Product link:** <offline metric> to <product metric> via <mechanism>
- **Rollout:** shadow <duration>, canary <n>%, abort if <condition>
- **Monitors:** input drift <how>, prediction drift <how>, product metric <how>
- **Rollback:** to v<n-1> via <mechanism>, takes <time>, tested on <date>
Verification
- Baseline and model numbers come from the same holdout; both quoted.
- Every holdout row is later in time than every training row.
- Shadow and production scores were compared on the same requests, and the skew is a number.
- Drift monitors were run against deliberately shifted input and fired.
- The rollback was executed once outside production and timed.
Red flags
| Thought | Reality |
|---|---|
| "0.97 on the first attempt" | Leakage. Find it before telling anyone the number. |
| "The baseline is too trivial to measure" | Then it is cheap, and it is the bar your model has to clear. |
| "Shuffle split is standard" | Not on temporal data. You trained on the future. |
| "Serving recomputes features slightly differently" | That is skew, and it is the most common silent production failure. |
| "Accuracy is 94%" | On which slice? The worst one is the one people complain about. |
| "It is doing fine in production" | Measured how? Models degrade without raising an error. |
| "We can retrain if it drifts" | Retraining is not a rollback. What serves traffic meanwhile? |
| "The offline metric improved" | Linked to which product outcome, by what mechanism? |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |