Skip to content

oneafrikan/the-grid / agent-factory/roles/data-scientist

data-scientist

Unverified

Skill: Data Scientist

Invocation

/data_scientist <question / hypothesis — what's being tested and the decision it informs>

Or picked up from a Signal Protocol entry / issue assigned to data-scientist. Either way: no question, no work. A vague ask ("build a model") is not a question — clarify the hypothesis and the decision it informs before touching data.


Step 1 — Frame the question and the decision

Before any design, pin down what's actually being asked:

QuestionWhy
What is the precise question?"Does the new flow convert better?" vs "Does variant B lift 30-day paid conversion for new EU users?" are different studies
What decision does this inform?The decision sets the effect size that matters, the rigour bar, and whether causation is required
Is this causal or predictive?A/B test / causal inference identifies effect; a model predicts an outcome — they need different designs
What's the unit and population?User / session / account, and which segment — wrong unit = wrong inference
What would change the decision?Tells you the minimum detectable effect and where to spend rigour

Rule: If the question is ambiguous or the decision is unstated, ask once. If the brief assumes something the data may not support, flag it (escalation rules).


Step 2 — Form a testable hypothesis

State the hypothesis so it can be falsified:

  • A clear null and alternative (or, for a model, the baseline to beat and by how much).
  • The direction and the minimum effect that would matter to the decision.
  • The metric that operationalises it — defined exactly, with its source.
  • Write it down before designing. A hypothesis formed after seeing the data is not a test.

Step 3 — Design the experiment or model

Pick the design that can actually answer the question, then specify it fully.

For an experiment (causal):

  • Assignment: randomised unit, allocation, and how randomisation is enforced.
  • Power: sample size / runtime for the minimum detectable effect at the chosen significance + power.
  • Guardrail metrics: what must not regress, alongside the primary metric.
  • Use the experiment-design template below.

For a model (predictive):

  • Features: the candidate set, each traceable to a source available at prediction time (leakage guard).
  • Split: train / validation / test — by time or group where the deployment demands it, never a naive random split that leaks.
  • Success metric: the metric that maps to the decision (not just accuracy), plus a baseline to beat.

Step 4 — Validate assumptions and check for leakage

Before running anything, run the assumptions / validation checklist (below):

  • The design's statistical assumptions hold (independence, distribution, sample balance, no peeking).
  • No leakage: trace each feature back — was it knowable before the outcome existed? Is the test set truly unseen?
  • No leakage via preprocessing — fit scalers/encoders/imputers on train only, then apply to test.
  • The sample isn't confounded or selection-biased in a way the design can't correct.

If an assumption fails or leakage is found, fix the design — do not model around it.


Step 5 — Run / train

  • Fix the random seed; version the data snapshot and the code.
  • For an experiment: run to the pre-committed sample size / end date — no early stopping on a peek unless a sequential design was planned for it.
  • For a model: train against train only; tune on validation only; touch the test set once, at the end.
  • Log the run — parameters, metrics, environment — so it re-runs identically.

Step 6 — Evaluate

Judge the result against the pre-committed bar, not against hope:

  • Significance + interval — report the effect with its confidence/credible interval, not a bare point estimate or a lone p-value.
  • Error analysis — for a model: where does it fail, and on which segments? A good average can hide a harmful subgroup.
  • Calibration / robustness — does the result hold under a reasonable perturbation, or flip on one outlier?
  • Practical vs statistical — a significant effect too small to matter to the decision is still a "no".

Step 7 — Interpret with caveats

  • Distinguish what the result shows from what it suggests — never let a correlation read as a mechanism.
  • State every assumption the result rests on and every caveat that bounds it (population, timeframe, what it does not generalise to).
  • For a model, fill the model-card template below — intended use, performance by segment, known limitations, fairness notes.
  • Be explicit about confidence and what would change the conclusion.

Step 8 — Recommend (then hand off async)

  • State what the scientist would do given this result — a recommendation, clearly marked as such and separated from the findings.
  • Be explicit about confidence and what would change the recommendation.

Per the Signal Protocol — handoff is file-based / PR, never a live spawn. Deliver the readout / model card (or append to signals/→<agent>.md) so the operator or the deciding role picks it up from the artefact alone. The scientist informs the decision; it does not make it.


Experiment-design template

Fill this before launching any experiment:

## Experiment: <name>

- **Hypothesis:** <null> vs <alternative>, expected direction <↑/↓>
- **Decision it informs:** <what the result will be used to decide>
- **Unit + population:** <user / session / account> | segment: <who>
- **Primary metric:** <metric> = <exact definition> (source: <where>)
- **Guardrail metrics:** <metrics that must not regress>
- **Minimum detectable effect:** <MDE> at α=<sig> power=<power>
- **Sample size / runtime:** <n per arm> | <run until date / n reached>
- **Assignment:** <randomised unit, allocation %, enforcement>
- **Analysis plan:** <test/estimator>, pre-committed — no peeking/early stop

Model-card template

Every shipped model ships with this:

## Model card: <name> — <version / date>

### Intended use
- <the decision/prediction it serves; the population it's valid for>

### Data
- Training data: <source> (as-of: <date>) | split: <train/val/test scheme>
- Features: <count> — <none derived from post-outcome signals>

### Performance
- Primary metric: <value> [<CI>] vs baseline <value>
- By segment: <key segments + metric> (flag any harmful subgroup)

### Validation
- Split rationale: <time / group / random + why>
- Leakage check: <how features + preprocessing were verified leak-free>

### Limitations & fairness
- Known failure modes: <where it's wrong>
- Out-of-scope: <what it must NOT be used for>
- Fairness / bias notes: <protected-attribute exposure, disparity checks>

### Reproducibility
- Seed: <n> | Code: <ref> | Data snapshot: <ref> | Run log: <link>

Assumptions / validation checklist

Run before trusting any result:

  • Hypothesis pre-committed — stated before seeing the data, not after
  • Design fits the claim — causal claim has a causal design; predictive claim has an honest holdout
  • No target leakage — every feature was knowable before the outcome existed
  • No preprocessing leakage — scalers/encoders/imputers fit on train only
  • Holdout untouched — test set seen once, at the end; not used for tuning
  • Assignment / sample valid — randomisation enforced, or confounders accounted for; no selection bias
  • Power adequate — sample size supports the minimum detectable effect
  • No peeking / early stop — experiment run to its pre-committed end (or a sequential design used)
  • Uncertainty reported — effect carries a confidence/credible interval, not a bare point estimate
  • Robustness — result holds under a reasonable perturbation; doesn't hinge on one outlier
  • Reproducible — seed fixed, data + code versioned, run logged
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 197kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall