khuynh22/agent-dev-team / skills/data-analysis
data-analysis
VerifiedData Analysis
The number is the smallest part of the answer. The definition, the denominator, the sanity check, and the caveat are what make it safe for somebody to act on.
Standing reference: references/data-quality-checklist.md.
Process
-
Restate the question and the decision behind it. "Did signups drop" and "should we roll back last night's release" need different analyses and different precision. Without the decision you cannot tell when the analysis is finished.
-
Define the metric in writing before you query. Numerator, denominator, time grain, time zone, exclusions: internal accounts, bots, refunds, test records. Most disputed numbers are two people using one word for two definitions, discovered an hour into an argument.
-
Verify the data before trusting it. Freshness for the period, row volume against the prior period, null rates on the columns you filter on. A pipeline gap is indistinguishable from a real drop until you check, and it is the more common cause.
-
Reconcile against something already believed. Billing, the previous dashboard, a hand count for one day. Quote both numbers. An unreconciled number is a draft.
-
Segment before concluding. A flat aggregate hides two populations moving in opposite directions, and the interesting finding is almost always inside a segment. Report sample size with every segment figure.
-
Rule out the boring explanations, in this order: a data problem, seasonality or day of week, a release, a tracking or instrumentation change, one large account, reporting lag. Name which you tested and how, including the ones that survived.
-
Separate correlation from cause, in the sentence itself. Observational comparisons between people who did and did not do something are selection effects until an experiment or a natural one says otherwise. Write "associated with", and say what would be needed to claim more.
-
State the caveat with the number, not underneath it. Sample size, window sensitivity, what would change the conclusion. A finding that only holds for one window is a finding about the window.
Experiment readouts
- The metric, the exposure definition, and the analysis window are fixed before looking.
- Report the absolute effect and the interval, not only whether it cleared a threshold.
- A non-significant result is a result. Report it as "no detectable effect above ", never as "no effect".
- Check the guardrail metrics and the sample ratio before reading the headline number.
Output
## Question: <what was asked, and the decision it feeds>
- **Metric:** <numerator> / <denominator>, <grain>, timezone <tz>, excluding <list>
- **Answer:** <number, absolute and relative change, n>
- **Query:** `<path>` — source <table>, as of <date>
- **Data checks:** freshness <ok | stale>, volume vs prior <n>, nulls <n>
- **Reconciled against:** <trusted source> — <both numbers>
- **Segments:** <segment> <value> (n=<n>) ...
- **Ruled out:** <data | seasonality | release | tracking change | outlier account>
- **Caveats:** <sample size, window sensitivity, what this cannot show>
- **Confidence:** high | medium | low, and what would raise it
Verification
- Every number traces to a saved, re-runnable query.
- Every rate has its denominator stated, and every segment has its sample size.
- The result reconciles against one independent source; both numbers are quoted.
- At least one alternative explanation was tested and reported, including when it survived.
- No causal verb appears where the design does not support one.
Red flags
| Thought | Reality |
|---|---|
| "Signups dropped 30%" | Over what base, over what window, and was the pipeline fresh? |
| "The correlation is strong" | Strong correlation on observational data is still not a cause. |
| "This segment converts 4x better" | With n=11. Sample size comes before the multiple. |
| "The number looks about right" | Right against what? Reconcile it. |
| "I will start the chart in March" | If the story needs March, the start date is the story. |
| "The dashboard says so" | The dashboard has a definition. Read it before repeating it. |
| "The average is flat" | Two populations moving opposite ways average to flat. Segment. |
| "Not significant, so no effect" | It means no effect detectable at this sample size. Say that. |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |