radarist/structured-analytic-skills / skills/quantitative-sanity-check
quantitative-sanity-check
VerifiedRecomputes the arithmetic that a document's own numbers imply — CAGR and compounding, price × volume unit economics, percentage vs percentage-point change, share-of-whole, survivorship and base-rate framing, Fermi order-of-magnitude — and flags every claim whose figures do not reproduce each other. Use when a report, pitch or press release states linked numbers — "$6B growing to $50B at a 25% CAGR", "40,000 customers at $99 a month is $120M ARR", "conversion improved 5%", "run a quantitative sanity check on these figures". Internal consistency only — not for checking a lone figure against outside sources (use `grounded-fact-check`) or sizing a market (use `estimate-market-size`).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 11.1 KB |
| agents/openai.yaml | config | 329 B |
| evals/evals.json | config | 5.7 KB |
| references/checks.md | markdown | 2.9 KB |
| scripts/sanity.py | script | 12.3 KB |
SKILL.md
View on GitHub---
name: quantitative-sanity-check
description: "Recomputes the arithmetic that a document's own numbers imply — CAGR and compounding, price × volume unit economics, percentage vs percentage-point change, share-of-whole, survivorship and base-rate framing, Fermi order-of-magnitude — and flags every claim whose figures do not reproduce each other. Use when a report, pitch or press release states linked numbers — \"$6B growing to $50B at a 25% CAGR\", \"40,000 customers at $99 a month is $120M ARR\", \"conversion improved 5%\", \"run a quantitative sanity check on these figures\". Internal consistency only — not for checking a lone figure against outside sources (use `grounded-fact-check`) or sizing a market (use `estimate-market-size`)."
license: MIT
metadata:
category: quantitative
method: Quantitative sanity check (internal-consistency arithmetic checklist)
origin: Bespoke checklist built on textbook conventions — Huff 1954; Wald 1943 via Mangel & Samaniego 1984; Miller 2015
version: "2.0.0"
---
# Quantitative Sanity Check
A quantitative sanity check recomputes, from a source's own figures, the numbers those figures imply — before anyone trusts, repeats or fact-checks them. Its principle is over-determination: start, end, period and growth rate fix each other; price, customers and revenue must multiply through. It is a bespoke checklist rather than one published method; each check applies a textbook convention — percentage versus percentage-point change (Huff, 1954; Miller, 2015), survivorship bias (Wald, 1943, via Mangel & Samaniego, 1984), base-rate neglect (Kahneman, 2011) and Fermi decomposition. It prevents repeating a headline number its own companions contradict.
## When to invoke
Invoke when:
- A document states two or more numbers that imply a third: "grows from $A to $B by {year} at X% CAGR", "{N} customers at ${P}/month generate ${R}".
- A change is reported without units — "improved 5%", "up 12 points" — and could be read either way.
- A "{X}% of successful {companies} did Y" statistic is about to be quoted, or a bare headline figure has a decomposable structure.
Do NOT invoke when:
- The claim is a single isolated number with nothing to check it against — `grounded-fact-check` verifies it externally.
- The question is whether a measured gap is real or noise — `test-significance`.
- The job is building a market estimate from scratch — `estimate-market-size`.
- The doubt is about a study's design, not its arithmetic — `assess-study-bias`.
## Procedure
Formulas, traps and tolerances: [references/checks.md](references/checks.md).
### 1 — Inventory the linked numbers
List every quantitative claim and mark the ones that are **linked** — presented as arithmetically related (start + end + rate; price + volume + total; share before + after + change; sensitivity + prevalence + "accuracy"). Only linked sets are checkable here; isolated figures need external verification. Record each claim verbatim; the verdict quotes it.
### 2 — Recompute compounding and CAGR
A (start, end, years, CAGR) quadruple is over-determined: recompute one member from the other three with `end = start × (1 + CAGR)^years` and `implied CAGR = (end/start)^(1/years) − 1`. Compounding periods = end year − start year (2024→2032 is 8, not 9); watch simple-versus-compound confusion and "doubling every N years" (CAGR = 2^(1/N) − 1). A gap beyond the claim's own precision is inconsistent — tolerance is half the last written digit, floored at ±0.2 pp, so a claim of "30 %" tolerates ±0.5 pp while "29.7 %" tolerates ±0.05 pp. Rounding is not a contradiction.
### 3 — Multiply through the unit economics
A (price, volume, total) triple must reproduce: `implied total = volume × price per period × periods per year` (or margin × volume for a contribution claim). Accept it within about ±20 % — tiers, churn timing and currency blur exact multiplication; beyond that the burden is on the source. State the gap as a multiple ("2.5× the recomputation") and name the benign explanations the source would owe.
### 4 — Separate percentage from percentage-point change
"Improved by 5%" from a 10 % baseline means either 10.5 % (relative) or 15 % (+5 percentage points) — a tenfold difference in implied impact. With before and after values, compute both readings and state which the prose matches. With only the delta, mark the claim **ambiguous**, report both readings and never silently choose the more impressive one. Reverse trap: "fell 50% to 20%" — from 40 % or from 70 %?
### 5 — Run the survivorship and base-rate checks
For "X% of {winners} did Y", ask for the inverted statistic — what fraction of *all* who did Y became winners? If only survivors were sampled, tag the claim survivorship-biased and treat it as anecdote. For a "99 % accurate" detection claim, recompute the positive predictive value at the stated prevalence, `PPV = sens × prev / (sens × prev + (1 − spec)(1 − prev))` — at 1 % prevalence a 99 %/95 % screen gives PPV ≈ 17 %.
### 6 — Cross-check a bare headline with a Fermi decomposition
For a headline with no companions but a decomposable structure, build one independent estimate — `quantity ≈ population × participation rate × frequency × unit value` — rounding each factor to the nearest half-order of magnitude. Within about 3× of the claim: plausible; 10× or more apart: implausible-as-stated — name the likely conflation. A market size worth two cited estimates goes to `estimate-market-size`.
### 7 — Render the verdict
Fill the output template. Each verdict is exactly one of **consistent** (reproduces within tolerance), **inconsistent** (the numbers contradict each other — show the recomputation), **ambiguous** (two readings — show both) or **unverifiable** (no linked numbers). Never resolve an inconsistency by picking the more plausible number; flag and re-source.
## Output template
```
## Quantitative sanity check — {document / claim set}
| # | Claim (verbatim) | Check | Recomputed | Verdict |
|---|------------------|-------|------------|---------|
| 1 | {claim} | {compounding | unit economics | pp vs % | base rate | Fermi} | {what the other numbers imply} | {consistent | inconsistent | ambiguous | unverifiable} |
**Overall:** {consistent | inconsistent | ambiguous | unverifiable}
**Action:** {pass through | flag in report | re-source via grounded-fact-check | drop claim}
**Tolerances used:** {CAGR ± half the claim's last written digit, min 0.2 pp; unit economics ±20 %; Fermi 3×}
```
Mandatory fields: every row's *Recomputed* and *Verdict*, plus *Overall* and *Action*. A verdict without recomputation is an opinion.
## Worked example
Illustrative investor memo for a fictional vendor, "Northwind Metrics" (figures invented). Rows 1–3 reproduced with `scripts/sanity.py` (each exits 1); row 4 by hand.
```
python3 scripts/sanity.py cagr --start 6.25 --end 50 --years 8 --claim 24.8 # implied CAGR 29.7% → INCONSISTENT
python3 scripts/sanity.py unit-econ --price 1188 --cost 0 --volume 40000 --claim-total 120000000 # 2.525x gap
python3 scripts/sanity.py pp --before 10 --after 15 --claim-pct 5 # +5pp, not 50% → UNIT CONFUSION
```
| # | Claim (verbatim) | Check | Recomputed | Verdict |
|---|------------------|-------|------------|---------|
| 1 | "$6.25B (2024) to $50B by 2032 at a 24.8% CAGR" | compounding | 8 periods → 29.7 % implied; 24.8 % implies $36.8B | inconsistent |
| 2 | "$120M ARR from 40,000 customers at $99/month" | unit economics | 40,000 × $1,188 = $47.5M; claim is 2.5× | inconsistent |
| 3 | "conversion improved 5%, from 10% to 15%" | pp vs % | +5 pp; relative change 50 % | ambiguous — restate as +5 pp |
| 4 | "screen is 99% accurate" (sens 99 %, spec 95 %, prevalence 1 %) | base rate | PPV = 0.0099 / 0.0594 = 16.7 % | inconsistent with "99 %" framing |
**Overall:** inconsistent. **Action:** flag rows 1, 2 and 4 for re-sourcing; restate row 3. The growth and revenue headlines cannot both be right.
## Verification
Before the verdict ships:
- [ ] Every inconsistent row shows the recomputation; every ambiguous row shows both readings.
- [ ] Compounding periods equal end year minus start year — recompute with `scripts/sanity.py cagr` (exit code 1 = flagged).
- [ ] Unit-economics rows state the period basis (monthly price × 12) and used the ±20 % tolerance.
- [ ] Rounding was not flagged as inconsistency ("$48M" for $47.5M is fine).
- [ ] No claim was "fixed" by choosing a side; survivorship-tagged claims name the missing denominator.
## Companion tool
`scripts/sanity.py` (stdlib only, Python 3.9+) recomputes steps 2–4: `cagr` (implied CAGR and multiple vs a claimed CAGR), `pp` (percentage-point and relative change, claim in either unit), `unit-econ` (margin, contribution vs a claimed total — for a revenue triple pass the annual price and `--cost 0`) and `share` (share-of-whole and reverse). Exit status: 0 consistent or no claim, 1 flagged, 2 usage error.
```bash
python3 scripts/sanity.py cagr --start 6.25 --end 50 --years 8 --claim 24.8
python3 scripts/sanity.py pp --before 5 --after 7 --claim-pct 2
python3 scripts/sanity.py unit-econ --price 99 --cost 40 --volume 40000 --claim-total 120000000
python3 scripts/sanity.py --selftest # 13 hand-checked cases, one PASS line each
```
Sample (`pp` command above):
```
before -> after : 5% -> 7%
absolute change : +2 percentage points
relative change : 40.0000%
UNIT CONFUSION : claim of 2% matches the pp reading (+2pp), not the relative one (40.0000%) — restate as '+2 percentage points'
```
Usable without the tool — every check is one line of arithmetic; the tool removes slips.
## Pair with adjacent skills
- `grounded-fact-check` — external verification of numbers that pass the internal check; run this skill first, since an inconsistent claim has nothing to ground.
- `test-significance` — when the question shifts from "do these numbers agree?" to "is the gap noise?".
- `estimate-market-size` — two-sided sizing when a Fermi cross-check is not enough.
- `abstain-or-escalate` — when a load-bearing number is inconsistent and cannot be re-sourced.
## Anti-patterns
- Do **not** "fix" an inconsistent claim by picking the more plausible number. Flag it; re-sourcing is another skill's job.
- Do **not** confuse internal consistency with truth — consistent numbers can still be externally wrong.
- Do **not** apply false precision: one decimal place shows a contradiction.
- Do **not** flag rounding as inconsistency, nor let the ±20 % tolerance excuse a 2.5× gap; statistical tests belong to `test-significance`.
## Reference
- J. E. Miller, *The Chicago Guide to Writing about Numbers*, 2nd ed. Chicago: University of Chicago Press, 2015. ISBN 978-0-226-18577-4 — percentage vs percentage-point conventions.
- D. Huff, *How to Lie with Statistics*. New York: W. W. Norton, 1954 — baseline games and percentage traps.
- M. Mangel and F. J. Samaniego, "Abraham Wald's Work on Aircraft Survivability," *Journal of the American Statistical Association*, vol. 79, no. 386, pp. 259–267, 1984. doi:10.1080/01621459.1984.10478038 — Wald's 1943 survivorship-bias case.
- D. Kahneman, *Thinking, Fast and Slow*. New York: Farrar, Straus and Giroux, 2011, ch. 14–16 — base-rate neglect.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |