Skip to content

coreyhaines31/marketingskills / skills/attribution

attribution

Verified

When the user wants to work out which marketing drives revenue, or asks whether a channel is really working, which numbers to trust, or how to credit podcasts and dark social: attribution models, platforms that over-claim, MMM, incrementality and holdout tests, self-reported attribution. For tracking and UTMs, see analytics. For pixels and CAPI, see conversion-tracking. For CRM revenue, see revops. For AI search blind spots, see ai-seo.

references/measurement-paradigms.md

View on GitHub
# Measurement Paradigms — MTA vs. MMM vs. Incrementality

Attribution *models* (see `attribution-models.md`) split credit *within* your tracked data. They can't tell you what would have happened anyway. That's what these three paradigms are for — increasingly rigorous, increasingly expensive ways to get closer to causality. Use this reference to help a user pick, and to explain how a test *reads* (not how to run the statistics).

## The three, compared

| | **MTA** (multi-touch) | **MMM** (media mix modeling) | **Incrementality** (experiments) |
|---|---|---|---|
| **Approach** | Bottom-up: stitch user-level touches, apply a model | Top-down: regress outcomes vs. spend/factors over time | Controlled: withhold exposure from a group, measure the difference |
| **Answers** | "Which touchpoints are on converting journeys?" | "What's each channel's aggregate contribution?" | "Did this channel *cause* lift I wouldn't have gotten?" |
| **Granularity** | Per user, per touch | Per channel, per week | Per test (one channel/tactic at a time) |
| **Data needed** | Clean cross-device user-level tracking | 2–3 yrs weekly data + spend variation | Ability to withhold + enough volume for significance |
| **Reacts** | Real-time | Slowly (weeks/quarters) | Per test cycle |
| **Handles offline/brand** | No | Yes | Yes (if you can split exposure) |
| **Privacy-durable** | Weak (cookie/ID loss) | Strong (aggregate) | Strong (aggregate) |
| **Cost/effort** | Low–medium | High | Medium–high |

## MTA — multi-touch attribution

**What it is:** the user-level approach most marketers mean by "attribution" — join a person's touches, apply first/linear/position/data-driven credit.

**Where it shines:** day-to-day, tactical decisions. "Is this campaign showing up on converting journeys?" Fast, granular, cheap if your tracking exists.

**Why it's structurally weakening:** MTA depends on tracking one person across touches and devices, and that data keeps eroding — Safari/Firefox ITP-style cookie limits, iOS ATT, browser consent gating, ad blockers, and ordinary cross-device / in-app-browser behavior (research on mobile, buy on desktop). (Chrome's third-party-cookie deprecation was announced, then walked back, so "cookies are going away" is no longer the clean story — but everything else on that list already limits user-level tracking today.)

Click-based paths add a second bias: **impression channels (social, video, display) often never enter the journey** — a view with no click leaves no stitchable touch — so MTA systematically **over-weights demand-capture** (search, email, direct) and **under-weights demand-creation**. MTA doesn't announce the gap: it silently under-measures anything it can't follow (dumping it into direct) and over-measures what it *can* see. Treat MTA as **incomplete and directionally biased** — not a clean lower bound (its retargeting/branded numbers are often *over*-stated) — and never the sole basis for a big reallocation.

**Use it for:** ongoing optimization and trend-watching — never as the sole basis for a big budget reallocation.

## MMM — media/marketing mix modeling

**What it is:** a top-down statistical model (historically regression; modern open-source options like Meta's Robyn or Google's Meridian) that explains an outcome (revenue, signups) as a function of spend per channel plus controls (seasonality, promotions, price, macro). It never looks at individuals — it's all aggregate time-series, which is exactly why privacy changes don't touch it.

**What it uniquely gives you:**
- **Offline + brand + hard-to-track channels** — TV, podcasts, OOH, PR, organic — because it works on aggregate spend and outcomes, not clicks.
- **Diminishing returns / saturation curves** — where the next dollar in a channel stops paying off.
- **A portfolio view** — how the whole mix drives the outcome, not credit for one journey.

**What it costs and where it's weak:**
- Needs **2–3 years of weekly data** and **real variation in spend** — if you always spend the same on Meta, the model can't learn Meta's effect. You sometimes have to deliberately vary budgets to feed it.
- **Correlational and slow** — it sees what moved together historically; it reacts in quarters, not days. It can't tell you what to do with tomorrow's campaign.
- **Sensitive to specification** — garbage controls, garbage coefficients. It's a real modeling exercise, not a dashboard toggle.

**Use it for:** annual/quarterly budget allocation across a material, multi-channel (esp. offline-inclusive) spend. Validate its coefficients with incrementality tests — MMM says "Meta contributed X"; a holdout proves whether that's causal.

## Incrementality — the experiments

**What it is:** the only paradigm that measures *causality* directly. Split your audience into an **exposed** group and a **withheld/control** group; the difference in outcomes is the **incremental lift** — conversions you got *because of* the channel, not ones that would have happened anyway.

**Common designs:**
- **Geo holdout / geo-lift** — run the channel in some regions, hold it out of comparable ones; compare outcomes. The workhorse for channels you can't split at the user level. *(Meta's GeoLift, Google's geo experiments.)*
- **PSA tests** — the control group is shown a *public-service ad* in your ad's place, so both groups are equally targeted and "ad-exposed"; the only difference is whether they saw *your* ad. Isolates ad effect from audience-quality bias (the exposed group isn't just "people the algorithm judged likely to convert").
- **Ghost ads** — the platform runs the auction but **withholds your ad** from the control group and logs the impression that *would* have served; the control user may see another advertiser's ad (or organic inventory) instead of a PSA placeholder. You compare converters among would-have-been-exposed vs. actually-exposed. Cleaner and cheaper than PSAs (no wasted PSA spend), and the modern default where the platform supports it. (Logging would-have-served opportunities helps identify the relevant comparison set; it is separate from the intent-to-treat analysis principle.)
- **Intent-to-treat analysis** — compare outcomes by the groups participants were randomly assigned to, regardless of whether they actually saw an ad. It estimates the effect of assignment and preserves the randomized comparison; it is not a synonym for switching a channel off. See the [Cochrane explanation of assignment effects](https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-08).
- **On-off (pulse) tests** — switch a channel off for a defined window and examine total outcomes. A simple before/after comparison can be confounded by seasonality, demand changes, and other campaigns. Use randomized regions or time blocks and an appropriate analysis when feasible; [Google's geo-experiment research](https://research.google.com/pubs/archive/38355.pdf) describes a controlled advertising design. Account for carryover and spillover before interpreting the change as causal.
- **Holdout audiences** — withhold a random % from a retargeting or email program; the delta is the program's true lift.

### How to *read* a test (not run the stats)

You don't need to compute significance by hand, but you must read a result honestly:

1. **Lift = exposed rate − control rate.** If exposed geos converted at 4.2% and control at 3.6%, incremental lift is ~0.6pp — the rest of that 4.2% would have converted anyway. This is why last-click ROAS is almost always *overstated*: it counts the whole 4.2%.
2. **Check the confidence interval.** "5% lift, interval −2% to +12%" includes zero and losses: you can't claim a win. Compare the interval to the smallest lift that changes your decision; if it straddles it, the test was too small to decide.
3. **Watch for contamination.** Control users who were reached anyway (cross-device, spillover between geos) shrink the measured gap. A "no lift" result can be a leaky test, not a dead channel.
4. **Translate to a decision.** Incremental CPA = spend ÷ *incremental* conversions (not total). This is the number that should drive budget — and it's usually worse than the platform's reported CPA, which is the point.

**Use it for:** the highest-stakes questions and the tiebreakers — "does retargeting actually do anything?", "is branded-search paid just buying clicks we'd get free?", "which of our two biggest channels is really driving growth?" You can only test a few things at a time, so spend those tests on the decisions that matter most.

## Putting them together (the mature stack)

They're layers, not competitors:

- **MTA** for daily/weekly tactical optimization and trend-watching — cheap, granular, directional.
- **MMM** for quarterly/annual portfolio allocation across the full mix including offline — durable, holistic. Prefer **marginal ROI / saturation curves** over average platform ROAS when allocating the next dollar.
- **Incrementality** as the **calibration and tiebreaker** — the ground-truth that keeps MTA and MMM honest, run on your biggest bets.

### Triangulated overrides (making MTA "closer to reality")

When incrementality, surveys, and platform gaps all say the same channel is under-credited in click-MTA, **don't invent a platform-gap multiplier**. Document a rule:

1. **Channel-level override (reporting layer):** after MTA runs, shift a bounded share of conversions from Direct / branded search into the under-credited channel for the window you have evidence for (holdout dates, campaign flight). Keep the raw MTA table intact; decision dashboards read the adjusted table. Fast to ship; channel-level only.
2. **User-level insertion (advanced):** before MTA runs, add synthetic touchpoints (e.g. "YouTube impression") onto a subset of Direct journeys where survey match, geo-test membership, or platform exposure logs justify it — then re-score. Needs engineering; unlocks creative/audience granularity.

Start with one channel, one override, temporal bounds, and a conservative slice of measured lift (not 100% of a noisy point estimate). Expand only after stakeholders trust the first rule.

**Scale to the user:** most SMBs need good UTMs + last-non-direct + a self-reported survey + the occasional on/off test — not an MMM. Bring MMM in when offline/brand spend is material and MTA visibly can't see it. Bring in formal incrementality when a single channel's budget is big enough that being wrong about it is expensive. Match the rigor to the size of the decision.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 323kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 323kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 323kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall