Skip to content

0xF4ng/aether-growth-fieldwork / growth/experiment-results

experiment-results

Verified

Closes an experiment: reads results against the original spec, renders a Win / No Signal / Loss / Confounded verdict, extracts the learning (why, not what), and specifies the next action. Required when any experiment window closes. Connects back to /growth-experiment for reformulated experiments, or forward to scaling decisions when the verdict is Win.

Experiment Results

Before starting

Confirm before reading results:

  • Original experiment spec — hypothesis, success metric, baseline, and window must be available. Reading results without a pre-registered spec is HARKing.
  • Window status — has the full experiment window closed? If not, only directional reads are permitted.
  • Confounder events — any outages, concurrent experiments, external events, or tracking changes during the window?
  • Sample size — is the sample sufficient to detect the expected effect? (stated in spec; re-evaluate if actual sample differs significantly)

Contract

This skill guarantees:

  • A BLOCK is issued if the original spec is unavailable — results are never read without a pre-registered hypothesis
  • "No Signal ≠ Loss" rule is enforced in every ambiguous result
  • Learning extraction answers WHY (mechanism), not just WHAT (outcome)
  • Every analysis produces an archive entry — no experiment closes without a record
  • Confounder scan is run before scoring any result

Role: Impartial Analyst. You read results against the original spec — not against hope or pessimism. You enforce the "No Signal ≠ Loss" rule. You extract learnings about mechanism, not just outcome. You require a next action before closing every analysis.


Inputs

Required before proceeding:

  • Original experiment spec (hypothesis + success metric + baseline)
  • Results data (metric outcomes for the measured window)

If original spec is unavailable:

BLOCK. Do not reconstruct. Return:
"The original experiment spec is not available. Reading results without a
pre-specified hypothesis is HARKing (Hypothesizing After Results are Known)
— it produces post-hoc justification, not learning.

A reconstructed spec is not a valid spec. Anyone who saw the results before
writing the hypothesis cannot produce an unbiased success criterion.

Action: locate the original spec document. If it cannot be found:
  → Archive this experiment as UNREADABLE
  → Log the hypothesis and metric used (if known) for future reference
  → Do NOT render a WIN/LOSS verdict
  → The only permitted output is: 'Results exist but cannot be evaluated
     without the pre-registered spec. Archived as UNREADABLE.'"

Pre-read checks

Before scoring, apply these checks:

Check 1 — Window status

IF experiment window has NOT fully closed:
  → Directional read only. Label output clearly:
    "DIRECTIONAL READ — window not closed. Do not make scaling decisions on this data."

IF window has closed (or user indicates it has):
  → Full verdict.

Check 2 — Confounder scan

Before reading the data, check for events that could have compromised the result:

Confounder typeSignal
Platform outageTraffic drop, increased errors, partial service during experiment window
Concurrent experimentAnother test on the same audience during the same window
External eventCompetitor launch, industry news, seasonal effect
Measurement errorUTM break, tracking event misfired, analytics gap
Audience contaminationTest and control audiences overlapped
IF confounder detected → note it before scoring.
A confounded result must be labeled Confounded, not Win or Loss.

Verdict framework

Four verdicts. Apply the one that matches, in order.

VERDICT 1 — WIN
Conditions:
  Primary metric meets or exceeds the stated threshold.
  No critical confounders that invalidate the measurement.
  Sample size is sufficient to distinguish signal from noise (use practical judgment).

Example: "Hypothesis: ≥35% email open rate in 7 days vs 22% baseline.
          Result: 41% open rate over 7 days across n=3,400 sends.
          Verdict: WIN. Threshold met; sample is substantial."

---

VERDICT 2 — NO SIGNAL
Conditions:
  Result is in the right direction but below threshold, OR
  Sample size is insufficient to detect the effect with reasonable confidence, OR
  Result is within noise range of the baseline.

CRITICAL RULE: No Signal ≠ Loss.
  "We didn't prove it works" is not the same as "it doesn't work."
  Before recording No Signal, state:
    - What sample size or run time would be needed to detect the effect if it exists?
    - Is the cost of running a larger experiment worth the information?

Example: "Hypothesis: ≥35% open rate. Result: 26% over 7 days on n=180 sends.
          Verdict: NO SIGNAL. Direction is positive but sample too small to
          distinguish from noise. Need ~500 sends to have confidence. Worth rerunning."

---

VERDICT 3 — LOSS
Conditions:
  Primary metric is below threshold AND in the wrong direction (worse than baseline), OR
  Primary metric is substantially below threshold with sufficient sample.

Example: "Hypothesis: ≥35% open rate. Result: 18% over 7 days on n=2,200 sends.
          Verdict: LOSS. Significantly below baseline (22%). Moving in the wrong
          direction with sufficient sample."

---

VERDICT 4 — CONFOUNDED
Conditions:
  A confounder was identified (from Check 2) that makes the result uninterpretable.

Example: "Hypothesis: conversion rate improvement. Result: 52% drop in traffic
          during day 4-6 of a 7-day window due to a platform outage.
          Verdict: CONFOUNDED. Traffic drop invalidates conversion comparison.
          Rerun with controls is required."

Learning extraction

Required. One paragraph. This is the most important output of the analysis.

Not: "The experiment didn't work." (That's a result, not a learning.) Not: "Open rates increased." (That's what happened, not why.)

Yes: what do you now know about the mechanism that you did not know before?

Learning extraction must answer:
  (a) WHY did the result happen? (mechanism, not just outcome)
  (b) What does this change about the next experiment you'd run?
  (c) What assumption was confirmed or invalidated?

Examples:

STRONG LEARNING:
  "We learned that subject lines naming a specific framework (e.g. 'the dual-write
  pattern') outperform general benefit claims (e.g. 'migrate without downtime').
  This suggests our ICP is optimization-seeking (looking for a technique to steal)
  rather than outcome-seeking. The next experiment should test a different technique
  name framing."

WEAK LEARNING (not acceptable):
  "We learned that subject line matters."
  (Too vague. What about it matters? For which audience? How will you apply it?)

Next step by verdict

WIN:
  → Scale or extend.
  → Identify: what must be true to hold this result at 3× the audience/volume?
    Are there resource constraints? Audience exhaustion risks? Quality degradation risks?
  → If repeatable: operationalize (add to playbook, document the winning tactic)

NO SIGNAL:
  → Option A: Rerun with larger sample, tighter audience, or longer window — if
    cost of certainty < value of the information.
  → Option B: Kill if the cost of running a definitive test exceeds the value of
    knowing the answer. Not every question is worth answering precisely.
  → Return to /growth-experiment with updated scope.

LOSS:
  → Kill the specific tactic.
  → Decide: was the hypothesis wrong (wrong channel / wrong message / wrong audience)
    or was the tactic wrong (right hypothesis, wrong execution)?
  → Return to /growth-experiment with the updated hypothesis.

CONFOUNDED:
  → Rerun with controls to prevent the confounder.
  → Note the confounder in the experiment archive so it is not overlooked next time.
  → If the confounder is structural (e.g. seasonal effect), adjust the experiment window.

Archive entry (required — close every analysis with this)

## Experiment Archive Entry

[Date] | [Experiment name]
Verdict: [Win / No Signal / Loss / Confounded]
One-line learning: [One sentence — what you know now that you didn't before]
Next action: [Specific action]

This entry feeds the experiment history that is referenced in future sprints. Without it, learnings are lost.


Output format

## Experiment Results

**Experiment:** [Name]
**Hypothesis:** [Original hypothesis — copy from spec]
**Success metric:** [Metric + threshold + window]
**Baseline:** [Pre-experiment value]

### Pre-read checks
Window status: [Closed / Open — directional only]
Confounders: [None detected / List with descriptions]

### Verdict
[WIN / NO SIGNAL / LOSS / CONFOUNDED]

Evidence:
  Result: [Actual metric value]
  vs Baseline: [Baseline value]
  vs Threshold: [Threshold — was it met?]
  Sample: [N = ?]
  Directional: [Positive / Negative / Mixed]

### Learning
[One paragraph: why did this happen? What assumption was confirmed or invalidated?
What does this change about the next experiment?]

### Next action
[Specific action: scale / rerun with [change] / kill / rerun with controls]
[For No Signal: state the sample needed for a definitive answer]

### Archive entry
[Date] | [Experiment name] | Verdict: [verdict] | Learning: [one sentence] | Next: [action]

Connects to

Upstream: /growth-experiment — the spec this analysis reads against.

Downstream: /growth-experiment (Loop: Loss or Confounded → reformulate) or scaling/operationalization (Win).


Brain reads / writes

If a companion aether-growth-brain repo is connected:

Before reading:

  • Read experiments/experiment-log.md — locate the original experiment entry; confirm hypothesis and success metric as pre-registered

Brain write (required — close every analysis):

  • Update the experiment entry in experiments/experiment-log.md with:
    • verdict: WIN / NO-SIGNAL / LOSS / CONFOUNDED
    • one_line_learning: the mechanism learned (not the outcome)
    • next_action: scale / rerun with [change] / kill / rerun with controls
    • date_closed: today's date
  • For WIN verdicts: also write to playbooks/messaging.md if the winning tactic is a messaging insight worth operationalizing

Brain not connected: archive entry is the minimum equivalent; recommend maintaining a local experiment log.


Anti-patterns

Anti-patternWhy it failsFix
Reading results without original spec (HARKing)Post-hoc justification is not learning; cannot distinguish "it worked" from "it happened to go up"Block until original spec is located; archive as UNREADABLE if not found
Declaring Loss when the signal window was too shortA 7-day test on a 30-day metric is No Signal, not LossCheck window against activation/retention cycle before rendering verdict
Declaring Win on insufficient samplen=15 cannot be a WinState sample needed for confidence; render No Signal if sample is inadequate
Scaling a Confounded resultA confounded Win might be a real LossAlways rerun with controls before scaling
Learning "the experiment didn't work"That's a result, not a learningLearning must answer: WHY? What mechanism was confirmed or invalidated?
Not writing the archive entryLearnings are lost; same experiments get re-run in later sprintsArchive entry is mandatory — it is the primary deliverable of the analysis
Treating concurrent experiment contamination as valid dataTwo simultaneous tests on the same audience make both unreadableLabel as Confounded; rerun sequentially

Related skills

SkillWhen to use
growth/growth-experiment/SKILL.mdAfter LOSS or CONFOUNDED: reformulate the experiment
growth/funnel-audit/SKILL.mdIf LOSS verdict reveals the hypothesis was wrong: re-audit the funnel
growth/retention-analysis/SKILL.mdIf experiment targeted retention churn type: compare result against curve shape prediction

Validation criteria

  • Original spec reconstructed or confirmed
  • Confounder check performed and noted
  • One of four verdicts applied with conditions stated
  • "No Signal ≠ Loss" rule applied when relevant
  • Learning paragraph answers WHY, not just WHAT
  • Next action specified
  • Archive entry written

References & Sources

Tier 1:

  • RICE / ICE prioritization: practical vs statistical significance framework
  • growth-experiment-results (growth-skills v1.0, score 8/10): Win/No Signal/Loss/Confounded framework, learning extraction method, archive entry format
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall