Skip to content

florianbonnet14/ThePowerOfAnalytics_ClaudeSkills / influential-factors-detective

influential-factors-detective

Verified

Identify and analyze qualitative factors that influence KPIs but can't be measured as numbers. Use when users need to find non-quantifiable drivers affecting metrics (like copy quality, design aesthetics, timing), plan A/B tests for subjective elements, bridge the gap between data and design/content decisions, understand why quantitative analysis is incomplete, optimize messaging or UX elements, or identify the actual levers to pull for improving KPIs at the leaf nodes of a KPI tree.

FileKindSize
SKILL.mdskill md9.6 KB
influential-factor-detective-tutorial.mdmarkdown2.3 KB
references/case-studies.mdmarkdown16.8 KB
references/common-factors.mdmarkdown10.3 KB
references/templates.mdmarkdown6.7 KB

references/case-studies.md

View on GitHub
# Case Studies

Real-world examples of influential factor testing with results and learnings.

## Case Study 1: Email Subject Lines

**Context:** B2B SaaS company, weekly product newsletter
**Audience:** 50,000 active users
**Baseline:** Generic topic-based subject lines
**Example:** "This Week's Product Updates"
**Baseline open rate:** 18%

### Approach
Testing different subject line strategies to improve open rates.

### Variations Tested

**Variant A: Question Format**
- Example: "Want to know what's new in [Product]?"
- Hypothesis: Questions engage curiosity
- Result: 16% open rate (-2pp) ❌
- Learning: Questions felt rhetorical, not compelling

**Variant B: Benefit-Focused**
- Example: "3 New Ways to Save Time This Week"
- Hypothesis: Clear value proposition drives opens
- Result: 24% open rate (+6pp) ✅
- Learning: Specific, quantified benefits resonate strongly

**Variant C: Urgency + Benefit**
- Example: "Don't Miss: 3 Time-Saving Features Added Today"
- Hypothesis: Urgency amplifies benefit
- Result: 21% open rate (+3pp) ⚠️
- Learning: Urgency felt salesy, reduced trust

**Variant D: Simple Personalization**
- Example: "[Name], Your Weekly [Product] Update"
- Hypothesis: Name personalization increases relevance
- Result: 19% open rate (+1pp) →
- Learning: Minimal impact, not worth implementation effort

### Winner
Variant B: Benefit-focused subject lines (+33% improvement)

### Key Learnings
1. **Specificity matters:** "3 Ways" outperformed "New Ways"
2. **Value over novelty:** Users care about outcomes, not just features
3. **Urgency backfires:** B2B audience sees it as manipulative
4. **Simple personalization has limited value:** Name-only insufficient
5. **Ongoing practice:** Now lead every subject line with specific benefit

### Follow-up Tests
- Test benefit quantification: "3 Ways" vs "Multiple Ways"
- Test benefit framing: Time vs money vs efficiency
- Test benefit specificity: Generic vs role-specific

---

## Case Study 2: Onboarding Flow Reduction

**Context:** Mobile fitness tracking app
**User base:** Growing consumer app, 100K MAU
**Baseline:** 8-step onboarding flow
**Baseline completion:** 42%
**Baseline Day 7 retention:** 35%

### Baseline Onboarding Steps
1. Personal info (name, email, age, weight, height)
2. Fitness goals (weight loss, muscle gain, maintenance, other)
3. Current fitness level (beginner, intermediate, advanced)
4. Preferred exercises (cardio, strength, flexibility, sports)
5. Workout schedule (days per week, preferred times)
6. Notification preferences (daily reminders, weekly summaries, achievements)
7. Connect wearable devices (Fitbit, Apple Watch, Garmin, etc.)
8. Try premium features (free trial prompt)

### Problem Identified
- 58% drop-off during onboarding
- User feedback: "Too many questions before I could try the app"
- Hypothesis: Friction prevents users from experiencing value

### Test Approach
Radically simplify onboarding to essential-only information.

### Variant: Minimal Onboarding (3 steps)
1. **Personal basics:** Name, email only
2. **Primary goal:** Single question with 4 clear options
3. **Quick start:** Immediate access to first workout

**Additional information collection:**
- Just-in-time prompts: Ask for workout schedule when scheduling first workout
- Progressive disclosure: Request device connection when user wants to import data
- Optional profile: Link to complete profile in settings (no push)
- Natural triggers: Notification preferences asked after first workout completion

### Results

| Metric | Baseline | Variant | Change |
|--------|----------|---------|--------|
| Onboarding completion | 42% | 78% | +36pp (+86%) ✅ |
| Day 7 retention | 35% | 48% | +13pp (+37%) ✅ |
| Premium conversion | 8% | 12% | +4pp (+50%) ✅ |
| Avg time to first workout | 8.5 min | 2.1 min | -6.4 min ✅ |

### Detailed Analysis

**Time-to-value impact:**
- Users starting first workout: 42% → 78%
- Average time from download to first workout completion: 8.5 min → 2.1 min
- Users completing 3+ workouts in first week: 18% → 31%

**Progressive disclosure success:**
- 67% of users eventually provided all information from old onboarding
- Information was provided when immediately relevant (not upfront)
- No complaints about repeated questions

**Premium conversion mechanics:**
- More users experiencing value → higher trial sign-ups
- Better quality trials (activated users vs drop-offs)
- Higher trial-to-paid conversion

### Key Learnings

1. **Time-to-value is critical:** Getting users to experience core value ASAP trumps data collection
2. **Completed onboarding ≠ activated user:** Old metric optimized wrong thing
3. **Users will provide info when needed:** Progressive disclosure works when contextual
4. **Friction compounds:** Each additional step has exponential impact on drop-off
5. **First impression matters:** Fast positive experience drives retention

### Implementation Changes

**New onboarding philosophy:**
- Minimum viable onboarding: Only info essential for first session
- Just-in-time data collection: Ask when you need it, when user sees value
- No optional setup steps: Everything beyond essentials is progressive
- Optimize for time-to-first-success: Not completion percentage

**Applied to other flows:**
- Account creation simplified across product
- Feature setup flows rebuilt with progressive disclosure
- Settings organization: Essential vs advanced distinction

### Follow-up Tests
- Test 2-step vs 3-step onboarding
- Test goal selection format (4 options vs 8 options)
- Test immediate workout suggestion vs user choice
- Test onboarding personalization based on goal selection

---

## Case Study 3: Pricing Page Layout

**Context:** B2B SaaS collaboration platform
**Audience:** Teams of 5-50 people
**Traffic:** 10,000 pricing page visitors/month
**Baseline conversion:** 8% to paid plan

### Baseline Design
- Three pricing tiers side-by-side (horizontal comparison)
- Equal visual weight for all plans
- Feature comparison table below
- No guidance on which plan to choose

**Plans:**
- Starter ($49/mo)
- Professional ($149/mo)
- Enterprise ($349/mo)

**Baseline distribution:**
- 40% chose Starter
- 35% chose Professional
- 25% chose Enterprise

### Problem Identified
- High bounce rate on pricing page (67%)
- User research: "Not sure which plan is right for us"
- Support tickets: "What's the difference between plans?"
- Hypothesis: Choice paralysis and lack of guidance reducing conversions

### Test Approach
Add visual hierarchy and recommendation to reduce decision friction.

### Variant: Highlighted Recommended Plan

**Visual changes:**
- Professional plan card 10% larger
- Different background color (light blue vs white)
- "Most Popular" badge at top
- "Recommended for most teams" subtext
- Subtle shadow effect for depth

**No changes to:**
- Pricing
- Features
- Copy (except recommendation text)
- Feature comparison table

### Results

| Metric | Baseline | Variant | Change |
|--------|----------|---------|--------|
| Overall conversion | 8.0% | 10.0% | +2.0pp (+25%) ✅ |
| Page bounce rate | 67% | 61% | -6pp ✅ |
| Time on page | 2.3 min | 1.8 min | -0.5 min ✅ |

**Plan Selection Shifts:**

| Plan | Baseline | Variant | Change |
|------|----------|---------|--------|
| Starter | 40% | 24% | -16pp |
| Professional | 35% | 58% | +23pp ✅ |
| Enterprise | 25% | 18% | -7pp |

### Revenue Analysis

**Per-visitor revenue:**
- Baseline: $105 average (8% × mix of plan values)
- Variant: $121 average (10% × higher share of mid-tier)
- **+15% revenue per visitor** ✅

**Why revenue increased more than conversion:**
- More conversions overall (+25%)
- Higher percentage on Professional plan (+66% of converters)
- Fewer on lower-tier Starter (-40% shift down)
- Slightly fewer on Enterprise (-28% shift from highest tier)

**Net effect:**
- Successfully guided more users to middle tier
- Reduced analysis paralysis (faster decisions)
- More predictable customer segmentation

### Detailed Learning

**Choice architecture matters:**
- Adding a recommendation doesn't feel pushy if sincere
- Visual hierarchy guides eye and decision
- "Most Popular" provides social proof
- Reduces cognitive load of comparing

**The "decoy effect":**
- Professional plan now serves as anchor
- Makes Starter seem more limited
- Makes Enterprise seem like overkill
- Comfortable middle ground

**Time-on-page reduction is positive:**
- Faster decisions, not fewer considerations
- Users found guidance helpful
- Less confusion, more confidence

**Business model implications:**
- Middle tier is sweet spot for:
  - Revenue per customer
  - Feature adoption
  - Growth potential (easier upsell to Enterprise)
  - Support burden (fewer basic plan users)

### Key Learnings

1. **Recommendation reduces friction:** Clear guidance helps users decide faster
2. **Visual hierarchy shapes choice:** Even subtle differences drive behavior
3. **Most Popular badge is powerful:** Social proof without being salesy
4. **Balanced portfolio shift:** Fewer low and high, more middle is ideal
5. **Faster decisions can be better decisions:** Reduced time-on-page with higher conversion

### Implementation Changes

**Broader application:**
- Feature comparison pages: Add "recommended" flags
- Product tours: Highlight most-used workflows
- Email templates: Suggest most popular
- Integration marketplace: Surface recommended integrations

**Testing philosophy shift:**
- Guidance is not manipulation if honest
- Users want help making decisions
- Reducing choice paralysis is user-friendly

### Follow-up Tests
- Test different badge text: "Most Popular" vs "Best Value" vs "Recommended"
- Test visual emphasis degree: Subtle vs dramatic
- Test adding brief plan selector quiz before pricing page
- Test enterprise plan repositioning (custom pricing vs fixed)

---

## Case Study 4: Landing Page Hero Image

**Context:** Marketing automation SaaS
**Campaign:** AdWords driving to feature-specific landing pages
**Traffic:** 5,000 visitors/month per landing page
**Baseline conversion:** 12% to free trial

### Test Setup
Testing hero image variations on landing pages while keeping all other elements constant.

### Variations Tested

**Control: Product Screenshot**
- Dashboard screenshot showing software interface
- Professional, feature-focused
- Result: 12.0% conversion (baseline)

**Variant A: Team Collaboration Photo**
- Diverse team in modern office, collaborating
- Lifestyle, aspirational
- Result: 10.5% conversion (-1.5pp) ❌

**Variant B: Individual Success Photo**
- Single professional looking satisfied at laptop
- Relatable, achievement-focused
- Result: 14.8% conversion (+2.8pp) ✅

**Variant C: Data Visualization**
- Abstract chart/graph with upward trends
- Results-oriented, sophisticated
- Result: 11.2% conversion (-0.8pp) →

### Winner
Variant B: Individual at laptop (+23% improvement)

### Deep-Dive Analysis

**Why team photo underperformed:**
- User testing revealed: "Looks like stock photo"
- Too corporate, not authentic
- Didn't connect to individual user's needs
- May have felt exclusionary (not everyone works in teams)

**Why individual photo worked:**
- Aspirational but relatable
- User saw themselves in the image
- Suggested personal productivity/success
- Authentic feeling (despite being stock)

**Why data viz was neutral:**
- Abstractness didn't add context
- Users wanted to see product or relate to person
- May have felt cold/impersonal

**Audience insight:**
- Marketing automation often purchased by individual marketers
- Even in teams, decision-maker is evaluating for personal use
- Personal success > team collaboration for this audience

### Key Learnings

1. **Match image to user's self-image:** Individual decision-makers relate to individual success
2. **Authenticity matters more than production value:** Real-feeling > polished
3. **Context clues affect trust:** Team photo triggered "stock photo" association
4. **Product screenshots can work but aren't always optimal:** Depends on product visual appeal
5. **Know your buyer:** Individual contributors relate to individual achievement

### Follow-up Tests
- Test individual photo variations (gender, age, setting diversity)
- Test product screenshot + person combo
- Test customer photos vs stock photos
- Test before/after visualization

---

## Case Study 5: Form Field Reduction

**Context:** White paper download page
**Company:** Enterprise software vendor
**Traffic:** 2,000 visitors/month
**Baseline conversion:** 24%

### Baseline Form (9 fields)
1. First Name
2. Last Name
3. Email
4. Company
5. Job Title
6. Company Size
7. Industry
8. Phone Number
9. Country

### Problem
Sales team wanted rich lead data, but marketing suspected form length hurting conversion.

### Test Approach
Progressive reduction to find optimal balance between conversion and lead quality.

**Variant A: 6 Fields (removed phone, company size, country)**
- Conversion: 31% (+7pp)
- Lead quality: 85% of baseline (sales feedback)
- Analysis: Good conversion lift, acceptable quality

**Variant B: 4 Fields (removed job title, industry too)**
- Conversion: 38% (+14pp)
- Lead quality: 70% of baseline (sales feedback)
- Analysis: High conversion, but sales complained about qualification

**Variant C: 3 Fields (only first name, email, company)**
- Conversion: 42% (+18pp)
- Lead quality: 60% of baseline (sales feedback)
- Analysis: Highest conversion, but significantly hurt lead qualification

### Progressive Profiling Implementation

**Winning approach: 4-field initial form + progressive profiling**

1. **Download page:** Name, Email, Company, Job Title (4 fields)
2. **Thank you page:** Optional: Phone, Company Size (2 fields, "Help us personalize")
3. **Future downloads:** Ask for missing fields (Industry, Country)
4. **Over time:** Build complete profile

**Results:**
- Initial conversion: 37% (+13pp vs baseline)
- Progressive data completion: 68% provide additional fields over time
- Complete profiles: 74% vs 24% baseline
- **Net improvement:** 3x more complete lead profiles

### Revenue Impact Analysis

**Baseline:**
- 2,000 visitors × 24% = 480 leads
- 480 leads × 85% qualified = 408 qualified leads
- Cost per qualified lead: $X

**Variant (4 fields + progressive):**
- 2,000 visitors × 37% = 740 initial leads
- 740 × 68% profile completion = 503 complete profiles
- 503 × 82% qualified = 412 qualified leads
- **+1% more qualified leads at +54% more raw leads**

### Key Learnings

1. **Friction compounds:** Each additional field has exponential impact
2. **Progressive profiling works:** Users will provide info over multiple touchpoints
3. **Context matters for data requests:** After value delivered, willingness increases
4. **Balance conversion and quality:** Raw conversion ≠ business outcome
5. **Timing of ask is critical:** Spread data collection across journey

### Implementation Changes
- All gated content moved to 4-field forms
- Thank you pages redesigned to request additional info
- Email nurture includes profile completion CTAs
- CRM enrichment for missing data points
- Sales given context on data completeness

### Follow-up Tests
- Test 3 vs 4 fields with aggressive progressive profiling
- Test value exchange messaging for additional fields
- Test mandatory vs optional field reduction
- Test field order optimization

---

## Common Patterns Across Case Studies

### Pattern 1: Reduce Friction
- Shorter subject lines, fewer form fields, minimal onboarding
- Every additional element = exponential drop-off
- Ruthlessly prioritize essential-only

### Pattern 2: Guide Choices
- Recommendations reduce analysis paralysis
- Visual hierarchy shapes decisions
- Clear guidance feels helpful, not pushy

### Pattern 3: Time-to-Value
- Get users to success moment ASAP
- Defer non-essential information collection
- First impression disproportionately affects retention

### Pattern 4: Authenticity Over Polish
- Real-feeling beats perfectly produced
- Avoid obvious stock photo triggers
- Match audience's self-image

### Pattern 5: Progressive Disclosure
- Collect data when it's relevant
- Users provide info after experiencing value
- Spread information requests across journey

### Pattern 6: Test Radical Changes
- Incremental tests often show incremental results
- Bold hypotheses drive breakthrough insights
- Don't be afraid to test 50%+ reductions

---

## Using These Case Studies

**For your own tests:**
1. **Start with proven patterns:** These case studies show what's possible
2. **Adapt to your context:** Your audience, product, and business are unique
3. **Test boldly:** Don't just test minor tweaks
4. **Measure what matters:** Conversion is vanity, revenue/retention is sanity
5. **Document learnings:** Build institutional knowledge

**When results differ:**
- Context matters more than best practices
- Your audience may behave differently
- Document why results differed
- Build your own pattern library

**Building on these learnings:**
- Combine winning patterns (e.g., form reduction + progressive profiling)
- Test interactions between factors
- Apply learnings across similar experiences
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall