Skip to content

DAAF-Contribution-Community/daaf / .claude/skills/data-scientist

data-scientist

Verified

Data science methodology and method-selection routing for quantitative research. Covers EDA, data validation, descriptive analysis, causal inference (IV, DiD, RD, synthetic control), clustering/PCA/UMAP, supervised ML, geospatial analysis, network analysis, and visualization design. Contains the canonical method-to-library routing tree, routed by execution language — Python: statsmodels (OLS/GLM/time series), pyfixest (FE/IV/DiD), linearmodels (RE/GMM/SUR), svy (complex surveys), scikit-learn (clustering/prediction ML), geopandas (spatial), igraph (network analysis); R: r-stats (OLS/GLM/time series), fixest (FE/IV/DiD), plm (panel RE/between, dynamic GMM), survey-r (complex surveys), tidymodels (ML), sf-terra (spatial), igraph-r (network analysis). Load the routed tool-specific skill before giving tool-specific advice or writing code — library skills encode environment constraints and curated caveats absent from general knowledge.

references/data-documentation.md

View on GitHub
# Data Documentation

Understanding existing documentation and creating documentation when none exists.

## Contents

- [Questions to Ask About Data](#questions-to-ask-about-data)
- [Understanding Columns](#understanding-columns)
- [Data Provenance](#data-provenance)
- [Working with Undocumented Data](#working-with-undocumented-data)
- [Creating Documentation](#creating-documentation)
- [Red Flags to Surface](#red-flags-to-surface)

## Questions to Ask About Data

Before any analysis, seek answers to these questions.

### About Data Source

| Question | Why It Matters |
|----------|----------------|
| Where does this data come from? | Determines reliability, update frequency |
| What system/process generated it? | Understanding generation can reveal biases |
| Who owns or maintains the source? | Knowing who to ask for clarification |
| When was it last updated? | Staleness affects relevance |
| How is data extracted/exported? | Export process might introduce issues |

### About Data Collection

| Question | Why It Matters |
|----------|----------------|
| What was the collection method? | Survey, sensor, transaction, manual entry each have biases |
| What time period does it cover? | Affects generalizability |
| Are there known gaps in coverage? | Missing data might not be random |
| What population does it represent? | Sampling bias affects conclusions |
| Was any filtering applied before export? | May be missing important subsets |

### About Data Quality

| Question | Why It Matters |
|----------|----------------|
| Are there known quality issues? | Avoid rediscovering known problems |
| How are errors handled? | Might be silently dropped or flagged |
| What validation exists at entry? | Determines trust in values |
| Has data been cleaned already? | Avoid double-cleaning |
| Are there special codes for missing/invalid? | -999, "N/A", etc. need handling |

### About Business Context

| Question | Why It Matters |
|----------|----------------|
| What business process generates this data? | Context helps interpret values |
| What decisions are made using it? | Understand stake and scrutiny level |
| Who are the domain experts? | Know who to consult |
| Are there regulatory requirements? | May constrain what you can do |
| What would be surprising in this data? | Domain knowledge catches errors |

## Understanding Columns

For each column, understand these attributes:

### Essential Attributes

| Attribute | Description | Example |
|-----------|-------------|---------|
| **Name** | Column identifier | `customer_id` |
| **Description** | What it represents in business terms | "Unique identifier assigned when customer account created" |
| **Data type** | Technical type | Integer, String, Datetime |
| **Nullability** | Can it be missing? | "Yes, for anonymous visitors" |
| **Domain/Valid values** | Allowed values | "Positive integers", "USA/CAN/MEX" |

### Extended Attributes (When Available)

| Attribute | Description |
|-----------|-------------|
| **Source** | Where the value originates |
| **Calculation logic** | If derived, how it's computed |
| **Update frequency** | How often it changes |
| **Dependencies** | Related columns or external data |
| **Known issues** | Documented problems or caveats |

### Questions for Each Column

When documentation is sparse, ask:

1. **What does this column represent in the real world?**
2. **What unit is it measured in?** (dollars, cents, kilograms, count)
3. **What does a null value mean?** (missing, not applicable, unknown)
4. **What are the valid values?** (range, enumeration, format)
5. **Is it directly captured or derived?** (if derived, from what?)
6. **Can it change over time?** (historical vs. current state)

## Data Provenance

Understanding where data comes from and how it got here.

### Lineage Questions

```
Data Flow:
Source System → Extraction → Transformation → Storage → Export → Your Analysis
     ↓              ↓             ↓            ↓          ↓
What system?   How often?    What changes?   What format?  Any filters?
```

### What to Document

1. **Source system(s)**: Original database, API, file system
2. **Extraction method**: SQL query, API call, file transfer
3. **Transformations applied**: Joins, filters, aggregations, cleaning
4. **Storage location**: Data warehouse, data lake, local file
5. **Export process**: How you received the data

### Why Provenance Matters

| Provenance Issue | Risk |
|-----------------|------|
| Unknown source | Can't validate correctness |
| Undocumented transforms | May have hidden assumptions |
| Multiple sources joined | Possible key mismatches |
| Old extraction | Data may be stale |
| Filtered export | Missing important subsets |

## Working with Undocumented Data

When you receive data with no documentation, follow this process.

### Step 1: Generate Basic Profile

**Python:**
```python
import polars as pl

# Generate documentation from data inspection: one row per column
profile_rows = []
for col in df.columns:
    profile_rows.append({
        "column": col,
        "dtype": str(df[col].dtype),
        "null_count": df[col].null_count(),
        "null_pct": round(df[col].null_count() / len(df) * 100, 1),
        "unique_count": df[col].n_unique(),
        # Joined to a string, as in the R version: columns differ in type, so a
        # list column of raw values would not share one dtype across rows
        "sample_values": ", ".join(
            str(v) for v in df[col].drop_nulls().unique(maintain_order=True).head(5).to_list()
        ),
    })

profile = pl.DataFrame(profile_rows)
print(profile)
```

**R:**
```r
library(dplyr)
library(tibble)

# Generate documentation from data inspection
profile <- tibble(
  column = names(df),
  dtype = sapply(df, \(x) paste(class(x), collapse = "/")),
  null_count = sapply(df, \(x) sum(is.na(x))),
  null_pct = round(sapply(df, \(x) sum(is.na(x))) / nrow(df) * 100, 1),
  unique_count = sapply(df, n_distinct),
  sample_values = sapply(df, \(x) paste(head(unique(na.omit(x)), 5), collapse = ", "))
)

print(profile)
```

### Step 2: Identify Granularity

**Critical question: What does each row represent?**

**Python:**
```python
# Test candidate keys for uniqueness -- identify what uniquely identifies each row
candidates = []

# Test single columns
for col in df.columns:
    if df[col].n_unique() == len(df):
        candidates.append([col])

# Test date + entity patterns
date_cols = [c for c in df.columns if "date" in c.lower() or "time" in c.lower()]
# A column matching both patterns (e.g. "update_time_id") stays a date column:
# selecting it twice would give a duplicate output name, which Polars rejects
id_cols = [c for c in df.columns if "id" in c.lower() and c not in date_cols]

for date_col in date_cols:
    for id_col in id_cols:
        if df.select([date_col, id_col]).n_unique() == len(df):
            candidates.append([date_col, id_col])

print("Candidate unique keys:")
for key in candidates:
    print(f"  {key}")
```

**R:**
```r
# Test candidate keys for uniqueness -- identify what uniquely identifies each row
candidates <- list()

# Test single columns
for (col in names(df)) {
  if (n_distinct(df[[col]]) == nrow(df)) {
    candidates <- c(candidates, list(col))
  }
}

# Test date + entity patterns
date_cols <- names(df)[grepl("date|time", names(df), ignore.case = TRUE)]
id_cols <- names(df)[grepl("id", names(df), ignore.case = TRUE)]

for (date_col in date_cols) {
  for (id_col in id_cols) {
    unique_count <- df |> distinct(across(all_of(c(date_col, id_col)))) |> nrow()
    if (unique_count == nrow(df)) {
      candidates <- c(candidates, list(c(date_col, id_col)))
    }
  }
}

cat("Candidate unique keys:\n")
for (key in candidates) {
  cat("  [", paste(key, collapse = ", "), "]\n")
}
```

### Step 3: Infer Column Meanings

**Python:**
```python
# Infer what each column might represent
for col in df.columns:
    inferences = []
    n_unique = df[col].n_unique()
    pct_unique = n_unique / len(df) * 100
    sample_vals = df[col].drop_nulls().unique(maintain_order=True).head(5).to_list()

    # Name-based inference
    name_lower = col.lower()
    if "id" in name_lower:
        inferences.append("Likely identifier/key")
    if "date" in name_lower or "time" in name_lower:
        inferences.append("Likely temporal")
    if "amt" in name_lower or "amount" in name_lower or "price" in name_lower:
        inferences.append("Likely monetary value")
    if "count" in name_lower or "num" in name_lower or "qty" in name_lower:
        inferences.append("Likely count/quantity")
    if "flag" in name_lower or "is_" in name_lower or "has_" in name_lower:
        inferences.append("Likely boolean indicator")

    # Cardinality-based inference
    if pct_unique > 95:
        inferences.append("High cardinality (ID? Free text?)")
    elif n_unique == 2:
        inferences.append("Binary (boolean?)")
    elif pct_unique < 1:
        inferences.append("Low cardinality (categorical)")

    print(f"\n{col} ({df[col].dtype}): {pct_unique:.1f}% unique")
    print("  Sample:", ", ".join(str(v) for v in sample_vals))
    if inferences:
        print("  Inferences:", "; ".join(inferences))
```

**R:**
```r
# Infer what each column might represent
for (col in names(df)) {
  inferences <- c()
  n_unique <- n_distinct(df[[col]])
  pct_unique <- n_unique / nrow(df) * 100
  sample_vals <- head(unique(na.omit(df[[col]])), 5)

  # Name-based inference
  name_lower <- tolower(col)
  if (grepl("id", name_lower)) inferences <- c(inferences, "Likely identifier/key")
  if (grepl("date|time", name_lower)) inferences <- c(inferences, "Likely temporal")
  if (grepl("amt|amount|price", name_lower)) inferences <- c(inferences, "Likely monetary value")
  if (grepl("count|num|qty", name_lower)) inferences <- c(inferences, "Likely count/quantity")
  if (grepl("flag|is_|has_", name_lower)) inferences <- c(inferences, "Likely boolean indicator")

  # Cardinality-based inference
  if (pct_unique > 95) inferences <- c(inferences, "High cardinality (ID? Free text?)")
  else if (n_unique == 2) inferences <- c(inferences, "Binary (boolean?)")
  else if (pct_unique < 1) inferences <- c(inferences, "Low cardinality (categorical)")

  cat(sprintf("\n%s (%s): %.1f%% unique\n", col, class(df[[col]]), pct_unique))
  cat("  Sample:", paste(sample_vals, collapse = ", "), "\n")
  if (length(inferences) > 0) cat("  Inferences:", paste(inferences, collapse = "; "), "\n")
}
```

### Step 4: Document As You Learn

As you discover what columns mean, CREATE documentation:

**Python:**
```python
# Build documentation dictionary as you learn
data_dictionary = {
    "customer_id": {
        "description": "Unique customer identifier",
        "type": "Integer",
        "nullable": False,
        "notes": "Assigned at account creation, never reused"
    },
    "order_date": {
        "description": "Date order was placed",
        "type": "Date",
        "nullable": False,
        "notes": "In UTC timezone"
    },
    # Add more as you discover...
}
```

**R:**
```r
# Build documentation as a tibble as you learn
data_dictionary <- tribble(
  ~column,        ~description,                  ~type,     ~nullable, ~notes,
  "customer_id",  "Unique customer identifier",  "integer", FALSE,     "Assigned at account creation, never reused",
  "order_date",   "Date order was placed",        "Date",    FALSE,     "In UTC timezone",
  # Add more as you discover...
)
print(data_dictionary)
```

## Creating Documentation

### Minimum Viable Data Dictionary

At minimum, document:

```markdown
## Data Dictionary: [Dataset Name]

### Overview
- **Source**: [Where data comes from]
- **Granularity**: [What each row represents]
- **Date range**: [Time period covered]
- **Row count**: [Number of records]

### Columns

| Column | Type | Description | Nullable | Notes |
|--------|------|-------------|----------|-------|
| customer_id | Int64 | Unique customer identifier | No | Primary key |
| order_date | Date | Date order was placed | No | UTC timezone |
| amount | Float64 | Order total in USD | No | Includes tax |
| status | String | Order status | No | Values: pending, shipped, delivered, cancelled |

### Known Issues
- [List any known data quality issues]

### Change Log
- [Date]: Initial documentation created
```

### Documentation Template

**Python:**
```python
# Generate a documentation template from the DataFrame
dataset_name = "Customer Orders"

template = f"""# Data Dictionary: {dataset_name}

## Overview
- **Source**: [TODO: Document source]
- **Granularity**: [TODO: What does each row represent?]
- **Date range**: [TODO: Time period covered]
- **Row count**: {len(df):,}
- **Column count**: {len(df.columns)}

## Columns

| Column | Type | Nullable | Unique | Description |
|--------|------|----------|--------|-------------|
"""

for col in df.columns:
    dtype = df[col].dtype
    nullable = "Yes" if df[col].null_count() > 0 else "No"
    unique_pct = round(df[col].n_unique() / len(df) * 100, 1)
    template += f"| {col} | {dtype} | {nullable} | {unique_pct}% | [TODO] |\n"

template += """
## Known Issues
- [TODO: Document known issues]

## Change Log
- [DATE]: Documentation created
"""

print(template)
```

**R:**
```r
# Generate a documentation template from data frame
dataset_name <- "Customer Orders"

cat(sprintf("# Data Dictionary: %s\n\n", dataset_name))
cat("## Overview\n")
cat("- **Source**: [TODO: Document source]\n")
cat("- **Granularity**: [TODO: What does each row represent?]\n")
cat("- **Date range**: [TODO: Time period covered]\n")
cat(sprintf("- **Row count**: %s\n", format(nrow(df), big.mark = ",")))
cat(sprintf("- **Column count**: %d\n\n", ncol(df)))
cat("## Columns\n\n")
cat("| Column | Type | Nullable | Unique | Description |\n")
cat("|--------|------|----------|--------|-------------|\n")

for (col in names(df)) {
  dtype <- class(df[[col]])
  nullable <- if (any(is.na(df[[col]]))) "Yes" else "No"
  unique_pct <- round(n_distinct(df[[col]]) / nrow(df) * 100, 1)
  cat(sprintf("| %s | %s | %s | %.1f%% | [TODO] |\n", col, dtype, nullable, unique_pct))
}

cat("\n## Known Issues\n- [TODO: Document known issues]\n\n")
cat("## Change Log\n- [DATE]: Documentation created\n")
```

## Red Flags to Surface

When reviewing data, flag these issues for discussion:

### Data Quality Red Flags

| Red Flag | Concern |
|----------|---------|
| >10% missing in critical column | May bias analysis |
| Unexpected data types | Possible parsing/export error |
| Values outside reasonable range | Data quality issue or different units |
| Dates in the future | Likely data entry errors |
| Duplicate primary keys | Data integrity violation |
| Inconsistent encodings | Same thing represented differently |

### Documentation Red Flags

| Red Flag | Concern |
|----------|---------|
| No documentation exists | Risk of misinterpretation |
| Documentation is outdated | Schema may have changed |
| Column meanings unclear | May use data incorrectly |
| No provenance information | Can't validate correctness |
| Unknown null conventions | May mishandle missing data |

### Sampling Red Flags

| Red Flag | Concern |
|----------|---------|
| Convenience sample | May not represent population |
| Low response rate | Non-response bias |
| Specific subsets only | Findings may not generalize |
| Time-limited data | Seasonality effects |
| Geographic limitations | Regional bias |

### Report Template for Red Flags

```markdown
## Data Quality Concerns

### Critical Issues
1. [Issue]: [Description and impact]
   - Recommendation: [How to address]

### Important Findings
1. [Finding]: [Description]
   - Implication: [What it means for analysis]

### Questions for Data Owner
1. [Question about unclear aspect]
2. [Question about data quality issue]

### Assumptions Made
1. [Assumption]: [Why it was necessary]
```
brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 321kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 197kUpdated Claude CodeCodexVerifiedInstall