Skip to content

brycewang-stanford/Auto-Empirical-Research-Skills / skills/25-HosungYou-Diverga/skills/universal-ma-codebook

universal-ma-codebook

Verified

Universal Meta-Analysis Codebook v2.2 - AI-Human collaboration for meta-analysis data extraction. 4-layer design: Identifiers, Statistics, AI Provenance, Human Verification. Integrates with C5/C6/C7 agents and Category I systematic review pipeline. Triggers: meta-analysis, codebook, data extraction, Hedges g, effect size

FileKindSize
SKILL.mdskill md14.2 KB
---
name: universal-ma-codebook
description: |
  Universal Meta-Analysis Codebook v2.2 - AI-Human collaboration for meta-analysis data extraction.
  4-layer design: Identifiers, Statistics, AI Provenance, Human Verification.
  Integrates with C5/C6/C7 agents and Category I systematic review pipeline.
  Triggers: meta-analysis, codebook, data extraction, Hedges g, effect size
version: "12.0.1"
---

# Universal Meta-Analysis Codebook

**Version**: 2.2
**Status**: Production
**Codex Review**: APPROVE WITH MINOR CHANGES (2026-01-26)
**Update**: Context-specific extensions (2026-01-26)

## Purpose

A **universal, AI-Human collaboration codebook** for meta-analysis that enables:
1. AI extraction from PDFs (RAG/OCR) with confidence tracking
2. Human verification of AI-extracted values
3. 100% human-verified data through structured workflow
4. Integration with Diverga C5/C6/C7 agents and Category I pipeline
5. **Context-specific extensions** for domain-specific moderator variables

## Context-Specific Extensions

The Universal Codebook supports **project-specific moderator layers** that extend the base 4-layer structure. Each meta-analysis context may have unique moderator variables.

### Extension Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│              UNIVERSAL CODEBOOK WITH CONTEXT EXTENSION              │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  LAYER 1: IDENTIFIERS + METADATA (10 fields) ← Universal           │
│  LAYER 2: CORE STATISTICAL VALUES (18 fields) ← Universal          │
│  LAYER 3: CONTEXT-SPECIFIC MODERATORS ← Project Extension          │
│  LAYER 4: AI EXTRACTION PROVENANCE ← Universal                     │
│  LAYER 5: HUMAN VERIFICATION (8 fields) ← Universal                │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

### Available Context Extensions

| Context | Extension File | Moderator Count |
|---------|----------------|-----------------|
| GenAI-HE | `GENAI_HE_CODEBOOK.md` | 15 moderators |
| Clinical Trials | `CLINICAL_CODEBOOK.md` | TBD |
| Educational Tech | `EDTECH_CODEBOOK.md` | TBD |

### Creating a Context Extension

1. **Define moderator variables** specific to your research domain
2. **Create classification rules** for categorical moderators
3. **Write AI extraction prompts** for each moderator
4. **Configure C6 agent** with the extension schema

```python
# Example: Configure C6 for GenAI-HE context
c6.configure_extension(
    context="genai_he",
    moderators=[
        {"name": "genai_tool", "type": "categorical", "values": ["ChatGPT", "Claude", ...]},
        {"name": "blooms_level", "type": "ordinal", "values": ["remember", "understand", ...]},
        {"name": "study_design", "type": "categorical", "values": ["RCT", "quasi", ...]},
    ],
    extraction_prompts=GENAI_HE_PROMPTS
)
```

### GenAI-HE Extension (Example)

**Layer 3: GenAI-HE Moderator Variables (15 fields)**

| Category | Fields |
|----------|--------|
| GenAI Tool | genai_tool, genai_tool_version, genai_access_type |
| Educational Outcome | blooms_level, outcome_dimension, learning_domain |
| Study Design | study_design, intervention_duration, intervention_type, control_condition |
| Context | education_level, discipline, country, sample_size_total, publication_type |

See: `GenAI-HE-Review-AIMC/docs/GENAI_HE_CODEBOOK.md` for full specification

## Architecture: Four-Layer Design

```
┌─────────────────────────────────────────────────────────────────────┐
│              UNIVERSAL META-ANALYSIS CODEBOOK v2.1                  │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  LAYER 1: IDENTIFIERS + METADATA (10 fields)                        │
│  study_id, es_id, citation, doi, year, design_type,                │
│  timepoint, arm_label_treat, arm_label_control, unit_of_analysis   │
│                                                                     │
│  LAYER 2: CORE STATISTICAL VALUES (18 fields)                       │
│  Primary: outcome_name → se_g (12)                                  │
│  Change-score: pre_mean_treat, pre_sd_treat, pre_post_corr (3)     │
│  Cluster: cluster_size, icc, n_clusters (3)                        │
│                                                                     │
│  LAYER 3: AI EXTRACTION PROVENANCE                                  │
│  Per-value: ai_value, source, method, confidence, derived_from     │
│  Stored as: ai_extraction_json                                     │
│                                                                     │
│  LAYER 4: HUMAN VERIFICATION (8 fields)                             │
│  verified_status, verified_by, verified_date, corrections_json,    │
│  disagreement_resolved, final_values_json, verification_notes,     │
│  sign_off                                                          │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

## Workflow: AI-Human Collaboration

### Phase 1: AI Extraction (Automated)

**Triggered by**: I3 RAG building completion or manual PDF upload

**Agent**: C6-DataIntegrityGuard

```python
# C6 extracts statistical values from PDFs
extraction_result = c6.extract_with_provenance(
    pdf_folder="./pdfs",
    methods=["rag", "ocr"],
    reconciliation="hierarchy",
    log_all_candidates=True
)
```

**Actions**:
1. I3 builds RAG from PDFs
2. C6 queries for statistical values (M, SD, n)
3. Multiple extraction methods run in parallel
4. Conflict resolution applied (hierarchy + tolerance)
5. Provenance recorded for all extractions
6. Hedges' g calculated where inputs complete

**Output**: All rows → `verified_status = PENDING`

### Phase 2: Triage (Automated)

**Agent**: C7-ErrorPreventionEngine

```python
# C7 categorizes by effective confidence
triage_result = c7.triage_extractions(
    data=extraction_result,
    thresholds=CONFIGURABLE_THRESHOLDS
)
```

**Categories**:
| Confidence | Status | Action |
|------------|--------|--------|
| HIGH (≥90%) | PROVISIONAL | Awaits sign-off |
| MEDIUM (70-89%) | PENDING | Recommended review |
| LOW (<70%) | PENDING | Required review (priority) |
| CONFLICT | PENDING | Required review (top priority) |

### Phase 3: Human Review (Mandatory)

**Interface**: Excel Review Queue or Web UI

**Critical Rule**: ALL rows require human verification

**Priority Queue**:
1. Conflicts detected (highest)
2. LOW confidence
3. MEDIUM confidence
4. HIGH confidence (spot check)

**Human Actions**:
- Verify AI extraction against PDF
- Correct errors, record reason
- Mark as VERIFIED or REJECTED
- Resolve conflicts

### Phase 4: Final Sign-Off

**Agent**: C5-MetaAnalysisMaster

```python
# C5 validates all gates pass
validation = c5.validate_final(
    data=verified_data,
    require_all_verified=True,
    require_all_signed_off=True
)
```

**Requirements**:
- All rows: `verified_status = VERIFIED`
- All rows: `sign_off = True`
- All gates pass (C5 validation)

**Result**: 100% Human-Verified Dataset

---

## Field Specifications

### Layer 1: Identifiers + Metadata

| Field | Type | Description | Example |
|-------|------|-------------|---------|
| `study_id` | str | Unique study identifier | "CHEN_2024" |
| `es_id` | str | Effect size ID | "CHEN_2024_01" |
| `citation` | str | Full APA citation | "Chen et al. (2024)..." |
| `doi` | str | DOI | "10.1000/xyz" |
| `year` | int | Publication year | 2024 |
| `design_type` | str | RCT\|QUASI\|PRE_POST | "RCT" |
| `timepoint` | str | Measurement timing | "post" |
| `arm_label_treat` | str | Treatment label | "ChatGPT group" |
| `arm_label_control` | str | Control label | "Traditional" |
| `unit_of_analysis` | str | individual\|cluster | "individual" |

### Layer 2: Core Statistical Values

#### Primary Statistics

| Field | Type | Required |
|-------|------|----------|
| `outcome_name` | str | Yes |
| `outcome_unit` | str | No |
| `es_type` | str | Yes |
| `analysis_type` | str | No |
| `n_treatment` | int | Yes |
| `n_control` | int | Yes |
| `m_treatment` | float | Conditional |
| `sd_treatment` | float | Conditional |
| `m_control` | float | Conditional |
| `sd_control` | float | Conditional |
| `hedges_g` | float | Derived |
| `se_g` | float | Derived |

#### Change-Score Fields (when es_type = CHANGE)

| Field | Type | Description |
|-------|------|-------------|
| `pre_mean_treat` | float | Pre-test mean |
| `pre_sd_treat` | float | Pre-test SD |
| `pre_post_corr` | float | Pre-post correlation (default 0.5) |

#### Cluster Fields (when unit_of_analysis = cluster)

| Field | Type | Description |
|-------|------|-------------|
| `cluster_size` | float | Average cluster size |
| `icc` | float | Intra-class correlation |
| `n_clusters` | int | Number of clusters |

### Layer 3: AI Extraction Provenance

Stored in `ai_extraction_json`:

```json
{
  "n_treatment": {
    "ai_value": 43,
    "source": "Table 2, p.8",
    "method": "OCR",
    "confidence": 85,
    "derived_from": null
  },
  "sd_treatment": {
    "ai_value": 12.5,
    "source": "Text p.11, 95% CI",
    "method": "CALCULATED",
    "confidence": 92,
    "derived_from": "CI_95: SE = (14.8-10.2)/3.92"
  }
}
```

### Layer 4: Human Verification

| Field | Type | Values |
|-------|------|--------|
| `verified_status` | str | PENDING\|PROVISIONAL\|VERIFIED\|REJECTED |
| `verified_by` | str | Reviewer initials |
| `verified_date` | date | Review date |
| `corrections_json` | json | {field: {ai_value, final_value, reason}} |
| `disagreement_resolved` | bool | Conflict resolved? |
| `final_values_json` | json | Human-confirmed values |
| `verification_notes` | str | Free text notes |
| `sign_off` | bool | Final approval |

---

## Confidence Thresholds (Configurable)

### Per-Field Thresholds

| Field | HIGH | MEDIUM | LOW |
|-------|------|--------|-----|
| n (sample size) | ≥95% | 80-94% | <80% |
| M (mean) | ≥90% | 70-89% | <70% |
| SD | ≥85% | 65-84% | <65% |
| hedges_g (derived) | ≥92% | 75-91% | <75% |
| se_g (derived) | ≥92% | 75-91% | <75% |
| pre_post_corr | ≥85% | 65-84% | <65% |
| icc | ≥80% | 60-79% | <60% |

### Per-Source Modifiers

| Source | Modifier |
|--------|----------|
| Structured table | +10% |
| Semi-structured figure | +5% |
| Unstructured text | 0% |
| Abstract only | -15% |
| OCR with artifacts | -20% |

**Formula**: `effective_confidence = base_confidence + source_modifier`

---

## Conflict Resolution

### Extraction Hierarchy

| Priority | Source | Weight |
|----------|--------|--------|
| 1 | Table cell | 1.0 |
| 2 | Figure data | 0.9 |
| 3 | In-text stats | 0.8 |
| 4 | Abstract | 0.5 |

### Tolerance Thresholds

| Value Type | Relative | Absolute |
|------------|----------|----------|
| n (sample size) | 5% | 2 |
| M (mean) | 10% | 0.5 |
| SD | 15% | 0.5 |

**Rule**: If disagreement exceeds EITHER threshold → Human review required

---

## Derived Value Verification

For calculated values (hedges_g, se_g), human verification means:

1. All source values (M, SD, n) verified
2. Formula appropriate for es_type
3. If change-score: pre_post_corr verified or documented default
4. If cluster: ICC and cluster_size verified
5. Final values recalculated from verified inputs

---

## Integration Points

### Systematic Review Pipeline Integration

```python
# After Stage 5 (RAG building)
# RAG query integration

rag = RAGQuery(project_path)
values = c6.extract_from_rag(
    rag=rag,
    fields=["n_treatment", "n_control", "m_treatment", "sd_treatment",
            "m_control", "sd_control"],
    fallback_to_ocr=True
)
```

### C5/C6/C7 Agent Roles

| Agent | Role in Codebook |
|-------|------------------|
| C5-MetaAnalysisMaster | Final validation, gate enforcement |
| C6-DataIntegrityGuard | Extraction, Hedges' g calculation |
| C7-ErrorPreventionEngine | Triage, conflict detection, warnings |

---

## Excel Template Structure

### Sheet 1: Codebook

One-page reference with field definitions

### Sheet 2: Data (Main)

41 columns (39 visible + 2 JSON)

### Sheet 3: Review Queue

Priority-ordered list of rows needing human review

### Sheet 4: Extraction Log

Audit trail of all AI extractions

---

## Success Metrics

| Metric | Target |
|--------|--------|
| AI extraction rate | ≥85% |
| AI confidence accuracy | ≥90% |
| Conflict detection rate | ≥95% |
| Human review completeness | 100% |
| Final sign-off rate | 100% |
| Data completeness (Hedges' g) | ≥95% |

---

## Commands

```bash
# Initialize codebook for a project
diverga codebook init --project genai-he

# Import AI extractions
diverga codebook import --source rag --project genai-he

# Generate review queue
diverga codebook queue --project genai-he

# Validate final dataset
diverga codebook validate --project genai-he
```

---

## References

- Plan document: `docs/plans/META_ANALYSIS_CODEBOOK_PLAN_V2.md`
- C5 agent: `.claude/skills/C5-meta-analysis-master/SKILL.md`
- C6 agent: `.claude/skills/C6-data-integrity-guard/SKILL.md`
- C7 agent: `.claude/skills/C7-error-prevention-engine/SKILL.md`
- Borenstein et al. (2021). Introduction to Meta-Analysis
- Cochrane Handbook Chapter 6: Extracting Data
- PRISMA 2020 Statement

---

*Created: 2026-01-26*
*Codex Review: APPROVE WITH MINOR CHANGES*
*Author: Claude Code*
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 323kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall