indranilbanerjee/digital-marketing-pro / skills/cro
cro
VerifiedPlan CRO for pages, forms and checkout with ICE-ranked fixes. One page → landing-page-audit. "reduce cart abandonment"
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 15.9 KB |
| ab-testing.md | markdown | 13.2 KB |
| checkout-optimization.md | markdown | 16.3 KB |
| form-optimization.md | markdown | 13.5 KB |
| landing-page-audit.md | markdown | 11.2 KB |
| personalization-testing.md | markdown | 14.4 KB |
| pricing-psychology.md | markdown | 13.7 KB |
personalization-testing.md
View on GitHub# Personalization Testing — Optimization & Experimentation Reference
A systematic framework for testing personalized experiences against one-size-fits-all approaches. Covers when to personalize, how to design segment-based experiments, holdout methodology, and measurement frameworks that isolate true incremental lift.
---
## Personalization vs Traditional A/B Testing
Not every page or experience benefits from personalization. Use this decision framework before investing in segment-specific variations.
### When to Use Standard A/B Testing (One-Size-Fits-All)
- Your audience is relatively homogeneous in intent and behavior
- You have a single dominant traffic source or user type
- You lack the traffic volume to split into meaningful segments (< 50K monthly visitors)
- The change you are testing is universally applicable (page speed, trust badges, form length)
- You are early in your optimization program and need foundational wins first
### When to Personalize
- Analytics show clear behavioral or demographic segments with different conversion patterns
- You have sufficient traffic to power tests within individual segments (each segment needs its own sample size calculation)
- Different traffic sources arrive with fundamentally different intent (branded search vs. cold display)
- Your product serves multiple distinct use cases or buyer personas
- You have the technical infrastructure to serve dynamic content reliably
### Decision Matrix
| Scenario | Approach | Rationale |
|----------|----------|-----------|
| Homepage for all traffic | Start with A/B, then personalize by source | Establish baseline before fragmenting |
| Pricing page | A/B test globally, then personalize by geography | Pricing sensitivity varies by market |
| Product page | Personalize by browse history / purchase history | Past behavior strongly predicts intent |
| Landing pages from paid ads | Personalize by ad group / keyword | Intent varies dramatically by search query |
| Email campaigns | Segment-based personalization from day one | Email lists have rich segmentation data |
---
## Segment-Based Testing
### Core Concept
Instead of testing Variant A vs Variant B across all visitors, test different variants within different segments simultaneously. Each segment gets its own experiment with its own control.
### Segment Definition Criteria
| Segment Dimension | Examples | Data Source |
|-------------------|----------|-------------|
| **Traffic source** | Organic, paid search, paid social, email, direct, referral | Analytics UTM parameters |
| **Device type** | Desktop, mobile, tablet | User-agent / device detection |
| **Geography** | Country, state/region, city, timezone | IP geolocation |
| **Visitor type** | New visitor, returning visitor, logged-in customer | Cookie / session data |
| **Funnel stage** | First visit, product viewer, cart abandoner, past purchaser | Behavioral tracking |
| **Engagement level** | Low (1 page), medium (2-4 pages), high (5+ pages or 3+ minutes) | Session analytics |
### Minimum Segment Size
Each segment must independently meet sample size requirements. A segment with 200 monthly visitors cannot power a meaningful test.
| Baseline CVR | MDE 20% Relative | MDE 30% Relative | MDE 50% Relative |
|---|---|---|---|
| 2% | 21,000 per variation | 9,800 per variation | 3,800 per variation |
| 5% | 8,200 | 3,800 | 1,500 |
| 10% | 3,800 | 1,800 | 680 |
| 20% | 1,700 | 770 | 290 |
*Values computed with `scripts/sample-size-calculator.py` (`--mde-type relative`, 95% significance, 80% power).*
**Rule of thumb:** If a segment cannot reach sample size within 6 weeks, merge it with an adjacent segment or test with a larger MDE.
---
## Behavioral Targeting Experiments
### Cart Value Thresholds
| Cart Value Tier | Personalized Experience | Hypothesis |
|-----------------|------------------------|------------|
| Below average ($0–$49) | Show free shipping threshold message: "Add $X more for free shipping" | Increases AOV by 15-25% |
| Average ($50–$99) | Show bundle recommendations: "Frequently bought together" | Increases items per order |
| Above average ($100+) | Show loyalty benefits: "You qualify for VIP free returns" | Reduces cart abandonment |
| High value ($250+) | Offer concierge chat or phone support | Reduces friction for high-stakes purchases |
### Browse History Personalization
- **Viewed category 3+ times, no purchase:** Show category-specific discount or social proof ("187 people bought this today")
- **Viewed product 2+ times:** Show urgency signal ("Only 3 left in stock") or price drop notification
- **Browsed comparison content:** Show comparison table or "why us" content on next visit
- **Read blog content only:** Show softer CTA (guide download) instead of hard CTA (buy now)
### Visit Frequency Experiments
| Visit Count | Visitor Type | Test Variations |
|-------------|-------------|-----------------|
| 1st visit | Explorer | Broad value proposition, educational content, social proof |
| 2nd–3rd visit | Evaluator | Feature comparisons, testimonials, case studies |
| 4th–6th visit | Deliberator | Risk reducers (guarantees, trials), urgency, direct CTA |
| 7+ visits | Stalled | Discount offer, live chat prompt, "still deciding?" email trigger |
---
## Dynamic Content Testing
### Headline Variations by Traffic Source
| Source | Headline Approach | Example |
|--------|-------------------|---------|
| **Branded search** | Product-focused, direct | "Start your free trial — no credit card required" |
| **Non-branded search** | Problem/solution match to keyword | "[Keyword pain point]? Here's how to fix it" |
| **Paid social** | Matches ad creative tone and offer | Mirror the exact promise from the ad |
| **Email** | Continuity with email subject line | Extend the narrative started in the email |
| **Referral** | Credibility from the referrer | "Recommended by [Referral Source] — see why" |
### CTA Variations by Funnel Stage
| Stage | Awareness | Consideration | Decision |
|-------|-----------|---------------|----------|
| **Primary CTA** | "Learn More" / "See How It Works" | "Compare Plans" / "View Demo" | "Start Free Trial" / "Buy Now" |
| **Secondary CTA** | "Download Guide" | "Talk to Sales" | "Get Custom Quote" |
| **Urgency layer** | None | "Limited beta spots" | "Offer ends [date]" |
### Pricing Display by Geography
- **Test by purchasing power:** Show localized pricing in local currency with purchasing-power-adjusted tiers
- **Test annual vs. monthly default:** Some markets respond better to monthly (lower sticker shock), others to annual (value-oriented)
- **Test payment methods:** Prominently display regionally preferred payment methods (BACS in UK, iDEAL in Netherlands, Pix in Brazil)
---
## Progressive Disclosure Testing
Test how much information to reveal at each interaction point.
### Information Layering Framework
| Layer | Content | Test Variables |
|-------|---------|----------------|
| **Layer 1 — Above fold** | Core value prop, primary CTA, hero visual | How much detail in initial view? |
| **Layer 2 — Scroll or click** | Features, social proof, supporting details | Accordion vs. full display vs. tabbed |
| **Layer 3 — Deep engagement** | Pricing, technical specs, comparison tables | Gate behind click vs. show immediately |
| **Layer 4 — Committed** | Checkout, form, account creation | Single-page vs. multi-step |
### Test Ideas by Disclosure Level
- **Short vs. long landing page:** Test a focused above-fold-only design against a long-form page for the same audience
- **Feature tours:** Inline feature tour (visible) vs. "See features" button (click to reveal)
- **Pricing visibility:** Show pricing on the landing page vs. "See pricing" link vs. require demo request
- **Form length:** Full form upfront vs. progressive form (name + email first, then details on next step)
---
## Holdout Testing for Personalization
### Why Holdouts Matter
Without a holdout group, you cannot measure whether personalization actually improves outcomes versus a well-optimized generic experience. Many personalization programs show apparent lifts that disappear when measured against a proper holdout.
### Holdout Design
| Component | Specification |
|-----------|---------------|
| **Holdout size** | 5–10% of total traffic (must be large enough to detect expected personalization lift) |
| **Assignment** | Random, cookie-based, persistent across sessions |
| **Duration** | Minimum 4 weeks; ideally ongoing |
| **Experience** | Holdout sees the best-performing generic version (not an unoptimized baseline) |
| **Measurement** | Compare personalized cohort aggregate performance vs. holdout aggregate |
### Holdout Metrics to Track
| Metric | Purpose |
|--------|---------|
| Conversion rate (personalized vs. holdout) | Core lift measurement |
| Revenue per visitor (personalized vs. holdout) | Ensures personalization drives revenue, not just clicks |
| Return visit rate | Does personalization improve retention? |
| Customer lifetime value (30/60/90 day) | Long-term impact beyond initial conversion |
| Segment-level performance | Which segments benefit most from personalization? |
### Interpreting Holdout Results
| Result | Interpretation | Action |
|--------|---------------|--------|
| Personalized > holdout by 5%+ (significant) | Personalization is delivering real value | Continue and expand |
| Personalized > holdout by 1-4% (not significant) | Marginal lift, may not justify complexity | Simplify or focus on highest-lift segments only |
| Personalized = holdout | Personalization adds complexity without value | Roll back; optimize the generic experience instead |
| Personalized < holdout | Personalization is actively harming performance | Diagnose immediately — likely over-segmentation or poor targeting |
---
## Personalization Pitfalls
| Pitfall | Description | Prevention |
|---------|-------------|------------|
| **Over-personalization** | So many segments that each receives a barely-tested experience | Cap active segments at 3-5 until each is validated with holdout data |
| **Filter bubbles** | Showing users only what they've already engaged with, limiting discovery | Include 10-20% "exploration" content in personalized feeds |
| **Privacy backlash** | Personalization feels invasive ("How do they know I looked at this?") | Use behavioral cues subtly; never expose the data you used to personalize |
| **Segment size too small** | Testing within segments that cannot reach statistical significance | Pre-calculate segment sample sizes; merge small segments |
| **Complexity creep** | Maintaining 15+ personalized experiences becomes unmanageable | Start simple; add complexity only when validated by holdout lift |
| **Stale personalization** | Rules based on outdated behavior or expired intent signals | Set expiration windows on behavioral data (7 days for browse, 30 days for purchase) |
| **Assuming personalization wins** | Deploying personalized experiences without testing against generic | Always run holdout tests before declaring personalization a success |
---
## Testing Roadmap — Maturity Progression
### Stage 1: Segmented A/B Testing (Months 1–3)
- Segment traffic by 2-3 major dimensions (source, device, new/returning)
- Run standard A/B tests within each segment
- Identify which segments behave differently
- **Goal:** Understand where one-size-fits-all fails
### Stage 2: Rule-Based Personalization (Months 4–6)
- Implement simple if/then rules: "If returning visitor from paid search, show Variant B"
- Test each rule against generic experience with holdout
- Build 3-5 validated personalization rules
- **Goal:** Prove personalization lift with holdout data
### Stage 3: Behavioral Personalization (Months 7–12)
- Layer in behavioral signals: browse history, engagement depth, cart behavior
- Build dynamic content blocks that respond to user behavior
- Expand holdout testing to measure cumulative personalization lift
- **Goal:** Personalization contributes measurable revenue lift
### Stage 4: Predictive Personalization (Year 2+)
- Use ML models to predict optimal experience per visitor
- Real-time content assembly based on predictive scores
- Continuous holdout testing with automated segment discovery
- **Goal:** Autonomous optimization at individual level
---
## Measurement Framework
### Core Personalization Metrics
| Metric | Formula | Target |
|--------|---------|--------|
| **Personalization lift** | (Personalized CVR - Holdout CVR) / Holdout CVR | > 5% to justify program |
| **Segment performance variance** | Std dev of CVR across active segments | Lower variance = better targeting |
| **Revenue per visitor delta** | Personalized RPV - Holdout RPV | Positive and statistically significant |
| **Personalization coverage** | % of traffic receiving a personalized experience | 60-90% (leave holdout + unmatched) |
| **Rule hit rate** | % of sessions matching at least one personalization rule | > 70% indicates good rule coverage |
### Segment Performance Comparison Template
```
SEGMENT: [Name]
TRAFFIC VOLUME: [Monthly visitors]
PERIOD: [Date range]
PERSONALIZED EXPERIENCE:
- Visitors: [N]
- Conversions: [N]
- CVR: [X%]
- RPV: [$X]
HOLDOUT (GENERIC):
- Visitors: [N]
- Conversions: [N]
- CVR: [X%]
- RPV: [$X]
LIFT: [X%] (p = [X], CI: [X% to X%])
REVENUE IMPACT: [$X incremental per month]
DECISION: [Expand / Maintain / Optimize / Retire]
```
---
## Tool Integration Reference
| Tool | Strength | Personalization Type | Price Tier |
|------|----------|---------------------|------------|
| **Optimizely** | Enterprise experimentation with advanced targeting | Rule-based + audience-based | Enterprise ($$$) |
| **VWO** | Visual editor + behavioral targeting | Rule-based + heatmap-informed | Mid-market ($$) |
| **Dynamic Yield** | AI-driven personalization + recommendations | Predictive + behavioral | Enterprise ($$$) |
| **LaunchDarkly** | Feature flag-driven personalization for product teams | Feature flags + progressive rollout | Mid-market ($$) |
| **Mutiny** | B2B website personalization by firmographic data | Account-based personalization | Mid-market ($$) |
| **Intellimize** | AI-powered, automatically tests combinations | Generative + predictive | Mid-market ($$) |
| **Convert** | Privacy-focused experimentation | Rule-based + audience-based | SMB–Mid ($–$$) |
| **Google Tag Manager + GA4** | Free audience-based content swaps | Basic rule-based (DIY) | Free |
**Integration requirements:** Any personalization tool needs clean data from your analytics platform, CRM, and CDP. Poor data quality makes personalization worse, not better — you end up targeting the wrong experiences to the wrong people.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |