Skip to content

radarist/structured-analytic-skills / skills/quantitative-sanity-check

quantitative-sanity-check

Verified

Recomputes the arithmetic that a document's own numbers imply — CAGR and compounding, price × volume unit economics, percentage vs percentage-point change, share-of-whole, survivorship and base-rate framing, Fermi order-of-magnitude — and flags every claim whose figures do not reproduce each other. Use when a report, pitch or press release states linked numbers — "$6B growing to $50B at a 25% CAGR", "40,000 customers at $99 a month is $120M ARR", "conversion improved 5%", "run a quantitative sanity check on these figures". Internal consistency only — not for checking a lone figure against outside sources (use `grounded-fact-check`) or sizing a market (use `estimate-market-size`).

FileKindSize
SKILL.mdskill md11.1 KB
agents/openai.yamlconfig329 B
evals/evals.jsonconfig5.7 KB
references/checks.mdmarkdown2.9 KB
scripts/sanity.pyscript12.3 KB
---
name: quantitative-sanity-check
description: "Recomputes the arithmetic that a document's own numbers imply — CAGR and compounding, price × volume unit economics, percentage vs percentage-point change, share-of-whole, survivorship and base-rate framing, Fermi order-of-magnitude — and flags every claim whose figures do not reproduce each other. Use when a report, pitch or press release states linked numbers — \"$6B growing to $50B at a 25% CAGR\", \"40,000 customers at $99 a month is $120M ARR\", \"conversion improved 5%\", \"run a quantitative sanity check on these figures\". Internal consistency only — not for checking a lone figure against outside sources (use `grounded-fact-check`) or sizing a market (use `estimate-market-size`)."
license: MIT
metadata:
  category: quantitative
  method: Quantitative sanity check (internal-consistency arithmetic checklist)
  origin: Bespoke checklist built on textbook conventions — Huff 1954; Wald 1943 via Mangel & Samaniego 1984; Miller 2015
  version: "2.0.0"
---
# Quantitative Sanity Check

A quantitative sanity check recomputes, from a source's own figures, the numbers those figures imply — before anyone trusts, repeats or fact-checks them. Its principle is over-determination: start, end, period and growth rate fix each other; price, customers and revenue must multiply through. It is a bespoke checklist rather than one published method; each check applies a textbook convention — percentage versus percentage-point change (Huff, 1954; Miller, 2015), survivorship bias (Wald, 1943, via Mangel & Samaniego, 1984), base-rate neglect (Kahneman, 2011) and Fermi decomposition. It prevents repeating a headline number its own companions contradict.

## When to invoke

Invoke when:

- A document states two or more numbers that imply a third: "grows from $A to $B by {year} at X% CAGR", "{N} customers at ${P}/month generate ${R}".
- A change is reported without units — "improved 5%", "up 12 points" — and could be read either way.
- A "{X}% of successful {companies} did Y" statistic is about to be quoted, or a bare headline figure has a decomposable structure.

Do NOT invoke when:

- The claim is a single isolated number with nothing to check it against — `grounded-fact-check` verifies it externally.
- The question is whether a measured gap is real or noise — `test-significance`.
- The job is building a market estimate from scratch — `estimate-market-size`.
- The doubt is about a study's design, not its arithmetic — `assess-study-bias`.

## Procedure

Formulas, traps and tolerances: [references/checks.md](references/checks.md).

### 1 — Inventory the linked numbers

List every quantitative claim and mark the ones that are **linked** — presented as arithmetically related (start + end + rate; price + volume + total; share before + after + change; sensitivity + prevalence + "accuracy"). Only linked sets are checkable here; isolated figures need external verification. Record each claim verbatim; the verdict quotes it.

### 2 — Recompute compounding and CAGR

A (start, end, years, CAGR) quadruple is over-determined: recompute one member from the other three with `end = start × (1 + CAGR)^years` and `implied CAGR = (end/start)^(1/years) − 1`. Compounding periods = end year − start year (2024→2032 is 8, not 9); watch simple-versus-compound confusion and "doubling every N years" (CAGR = 2^(1/N) − 1). A gap beyond the claim's own precision is inconsistent — tolerance is half the last written digit, floored at ±0.2 pp, so a claim of "30 %" tolerates ±0.5 pp while "29.7 %" tolerates ±0.05 pp. Rounding is not a contradiction.

### 3 — Multiply through the unit economics

A (price, volume, total) triple must reproduce: `implied total = volume × price per period × periods per year` (or margin × volume for a contribution claim). Accept it within about ±20 % — tiers, churn timing and currency blur exact multiplication; beyond that the burden is on the source. State the gap as a multiple ("2.5× the recomputation") and name the benign explanations the source would owe.

### 4 — Separate percentage from percentage-point change

"Improved by 5%" from a 10 % baseline means either 10.5 % (relative) or 15 % (+5 percentage points) — a tenfold difference in implied impact. With before and after values, compute both readings and state which the prose matches. With only the delta, mark the claim **ambiguous**, report both readings and never silently choose the more impressive one. Reverse trap: "fell 50% to 20%" — from 40 % or from 70 %?

### 5 — Run the survivorship and base-rate checks

For "X% of {winners} did Y", ask for the inverted statistic — what fraction of *all* who did Y became winners? If only survivors were sampled, tag the claim survivorship-biased and treat it as anecdote. For a "99 % accurate" detection claim, recompute the positive predictive value at the stated prevalence, `PPV = sens × prev / (sens × prev + (1 − spec)(1 − prev))` — at 1 % prevalence a 99 %/95 % screen gives PPV ≈ 17 %.

### 6 — Cross-check a bare headline with a Fermi decomposition

For a headline with no companions but a decomposable structure, build one independent estimate — `quantity ≈ population × participation rate × frequency × unit value` — rounding each factor to the nearest half-order of magnitude. Within about 3× of the claim: plausible; 10× or more apart: implausible-as-stated — name the likely conflation. A market size worth two cited estimates goes to `estimate-market-size`.

### 7 — Render the verdict

Fill the output template. Each verdict is exactly one of **consistent** (reproduces within tolerance), **inconsistent** (the numbers contradict each other — show the recomputation), **ambiguous** (two readings — show both) or **unverifiable** (no linked numbers). Never resolve an inconsistency by picking the more plausible number; flag and re-source.

## Output template

```
## Quantitative sanity check — {document / claim set}

| # | Claim (verbatim) | Check | Recomputed | Verdict |
|---|------------------|-------|------------|---------|
| 1 | {claim} | {compounding | unit economics | pp vs % | base rate | Fermi} | {what the other numbers imply} | {consistent | inconsistent | ambiguous | unverifiable} |

**Overall:** {consistent | inconsistent | ambiguous | unverifiable}
**Action:** {pass through | flag in report | re-source via grounded-fact-check | drop claim}
**Tolerances used:** {CAGR ± half the claim's last written digit, min 0.2 pp; unit economics ±20 %; Fermi 3×}
```

Mandatory fields: every row's *Recomputed* and *Verdict*, plus *Overall* and *Action*. A verdict without recomputation is an opinion.

## Worked example

Illustrative investor memo for a fictional vendor, "Northwind Metrics" (figures invented). Rows 1–3 reproduced with `scripts/sanity.py` (each exits 1); row 4 by hand.

```
python3 scripts/sanity.py cagr --start 6.25 --end 50 --years 8 --claim 24.8    # implied CAGR 29.7% → INCONSISTENT
python3 scripts/sanity.py unit-econ --price 1188 --cost 0 --volume 40000 --claim-total 120000000   # 2.525x gap
python3 scripts/sanity.py pp --before 10 --after 15 --claim-pct 5              # +5pp, not 50% → UNIT CONFUSION
```

| # | Claim (verbatim) | Check | Recomputed | Verdict |
|---|------------------|-------|------------|---------|
| 1 | "$6.25B (2024) to $50B by 2032 at a 24.8% CAGR" | compounding | 8 periods → 29.7 % implied; 24.8 % implies $36.8B | inconsistent |
| 2 | "$120M ARR from 40,000 customers at $99/month" | unit economics | 40,000 × $1,188 = $47.5M; claim is 2.5× | inconsistent |
| 3 | "conversion improved 5%, from 10% to 15%" | pp vs % | +5 pp; relative change 50 % | ambiguous — restate as +5 pp |
| 4 | "screen is 99% accurate" (sens 99 %, spec 95 %, prevalence 1 %) | base rate | PPV = 0.0099 / 0.0594 = 16.7 % | inconsistent with "99 %" framing |

**Overall:** inconsistent. **Action:** flag rows 1, 2 and 4 for re-sourcing; restate row 3. The growth and revenue headlines cannot both be right.

## Verification

Before the verdict ships:

- [ ] Every inconsistent row shows the recomputation; every ambiguous row shows both readings.
- [ ] Compounding periods equal end year minus start year — recompute with `scripts/sanity.py cagr` (exit code 1 = flagged).
- [ ] Unit-economics rows state the period basis (monthly price × 12) and used the ±20 % tolerance.
- [ ] Rounding was not flagged as inconsistency ("$48M" for $47.5M is fine).
- [ ] No claim was "fixed" by choosing a side; survivorship-tagged claims name the missing denominator.

## Companion tool

`scripts/sanity.py` (stdlib only, Python 3.9+) recomputes steps 2–4: `cagr` (implied CAGR and multiple vs a claimed CAGR), `pp` (percentage-point and relative change, claim in either unit), `unit-econ` (margin, contribution vs a claimed total — for a revenue triple pass the annual price and `--cost 0`) and `share` (share-of-whole and reverse). Exit status: 0 consistent or no claim, 1 flagged, 2 usage error.

```bash
python3 scripts/sanity.py cagr --start 6.25 --end 50 --years 8 --claim 24.8
python3 scripts/sanity.py pp --before 5 --after 7 --claim-pct 2
python3 scripts/sanity.py unit-econ --price 99 --cost 40 --volume 40000 --claim-total 120000000
python3 scripts/sanity.py --selftest        # 13 hand-checked cases, one PASS line each
```

Sample (`pp` command above):

```
before -> after : 5% -> 7%
absolute change : +2 percentage points
relative change : 40.0000%
UNIT CONFUSION  : claim of 2% matches the pp reading (+2pp), not the relative one (40.0000%) — restate as '+2 percentage points'
```

Usable without the tool — every check is one line of arithmetic; the tool removes slips.

## Pair with adjacent skills

- `grounded-fact-check` — external verification of numbers that pass the internal check; run this skill first, since an inconsistent claim has nothing to ground.
- `test-significance` — when the question shifts from "do these numbers agree?" to "is the gap noise?".
- `estimate-market-size` — two-sided sizing when a Fermi cross-check is not enough.
- `abstain-or-escalate` — when a load-bearing number is inconsistent and cannot be re-sourced.

## Anti-patterns

- Do **not** "fix" an inconsistent claim by picking the more plausible number. Flag it; re-sourcing is another skill's job.
- Do **not** confuse internal consistency with truth — consistent numbers can still be externally wrong.
- Do **not** apply false precision: one decimal place shows a contradiction.
- Do **not** flag rounding as inconsistency, nor let the ±20 % tolerance excuse a 2.5× gap; statistical tests belong to `test-significance`.

## Reference

- J. E. Miller, *The Chicago Guide to Writing about Numbers*, 2nd ed. Chicago: University of Chicago Press, 2015. ISBN 978-0-226-18577-4 — percentage vs percentage-point conventions.
- D. Huff, *How to Lie with Statistics*. New York: W. W. Norton, 1954 — baseline games and percentage traps.
- M. Mangel and F. J. Samaniego, "Abraham Wald's Work on Aircraft Survivability," *Journal of the American Statistical Association*, vol. 79, no. 386, pp. 259–267, 1984. doi:10.1080/01621459.1984.10478038 — Wald's 1943 survivorship-bias case.
- D. Kahneman, *Thinking, Fast and Slow*. New York: Farrar, Straus and Giroux, 2011, ch. 14–16 — base-rate neglect.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 196kUpdated Claude CodeCodexVerifiedInstall